Add tokenizer_data_gen to build process

[iluuu1994]

It happens on occasion that people forget to run tokenizer_data_gen.sh when updating the grammar. So this PR migrates the script to PHP and runs it automatically when zend_language_parser.y is changed.

Open questions

• Currently I'm only looking for the PHP binary in the build directory, the stubs also look for a global PHP binary. Not sure if that's necessary.

php-src/build/Makefile.global

Line 152 in dda0cea

if test `php -v | head -n1 | cut -d" " -f 2 | sed "s/$$/\n7.0.99/" | sort -rV | head -n1` != "7.0.99"; then \

• Should this be back-ported to PHP 7.3?

Advantages

• ext/tokenizer/tokenizer_data.c should never be out of sync again
• Runs on Windows

Disadvantages

• Might break some build tools/environments
• Unnecessary execution of the script when changing the parser in most cases

Feedback appreciated.

[codecov-io]

Codecov Report

Merging #6723 (dda0cea) into master (dda0cea) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6723   +/-   ##
=======================================
  Coverage   72.16%   72.16%           
=======================================
  Files         892      892           
  Lines      317896   317896           
=======================================
  Hits       229395   229395           
  Misses      88501    88501           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dda0cea...4b7510f. Read the comment docs.

[nikic]

I think we need to add detection of a PHP binary on the autoconf side. Using PHP_EXECUTABLE for this purpose doesn't seem right -- it might not yet exist, or be broken. We should be using system PHP, not the one we're building.

[iluuu1994]

or be broken

That's a good point. I wonder if we should also rethink using the freshly built PHP binary for the stubs then.

[cmb69]

Should this be back-ported to PHP 7.3?

7.3 is in security mode, so no. Might still be good for PHP 7.4, though.

[iluuu1994]

Thinking about it, it's probably unnecessary for 7.4/8.0 as patches will almost certainly not introduce changes to the grammar.

[nikic]

Please rebase over fa9e2b3 and use $(PHP) instead of $(PHP_EXECUTABLE).

[nikic]

Remove indentation here to keep this compatible with PHP < 7.3. We should probably bump our PHP version requirement, but this heredoc block probably isn't the right reason to do it :)

[iluuu1994]

Thanks Nikita!