Browse Source

fix: implement Unicode identifiers

According to cppreference, identifiers are `(XID_Start | '_')
XID_Continue*`, which is the case as of C++23 and C2x. I have confirmed
this myself with the drafts of C++23 and C2x.

https://en.cppreference.com/w/cpp/language/identifiers

Clang indeed implements identifiers as `(XID_Start | '_') XID_Continue*`
in C++ mode and C2x mode, with a slight extension to the character set
to include some extra math characters:
231992d9b8/clang/lib/Lex/Lexer.cpp (L1517-L1530)
pull/127/head
Jade Lovelace 3 years ago
parent
commit
93135e2eb5
4 changed files with 5751 additions and 85 deletions
  1. +1
    -1
      grammar.js
  2. +1
    -1
      src/grammar.json
  3. +5730
    -83
      src/parser.c
  4. +19
    -0
      test/corpus/expressions.txt

+ 1
- 1
grammar.js View File

@ -982,7 +982,7 @@ module.exports = grammar({
false: $ => token(choice('FALSE', 'false')),
null: $ => 'NULL',
identifier: $ => /[a-zA-Z_]\w*/,
identifier: $ => /(\p{XID_Start}|_)\p{XID_Continue}*/,
_type_identifier: $ => alias($.identifier, $.type_identifier),
_field_identifier: $ => alias($.identifier, $.field_identifier),

+ 1
- 1
src/grammar.json View File

@ -6425,7 +6425,7 @@
},
"identifier": {
"type": "PATTERN",
"value": "[a-zA-Z_]\\w*"
"value": "(\\p{XID_Start}|_)\\p{XID_Continue}*"
},
"_type_identifier": {
"type": "ALIAS",

+ 5730
- 83
src/parser.c
File diff suppressed because it is too large
View File


+ 19
- 0
test/corpus/expressions.txt View File

@ -52,6 +52,25 @@ int main() {
---
(translation_unit
(function_definition
(primitive_type)
(function_declarator (identifier) (parameter_list))
(compound_statement
(expression_statement (identifier))
(expression_statement (identifier)))))
============================================
Unicode Identifiers
============================================
int main() {
µs;
blah_accenté;
}
---
(translation_unit
(function_definition
(primitive_type)

Loading…
Cancel
Save