dnrops
/
antlr4
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge dev into master (#3642) 

* Get rid of reflection in CodeGenerator

* Rename TargetType -> Language

* Remove TargetType enum, use String instead as it was before

Create CodeGenerator only one time during grammar processing, refactor code

* Add default branch to appendEscapedCodePoint for unofficial targets (Kotlin)

* Remove getVersion() overrides from Targets since they return the same value

* Remove getLanguage() overrides from Targets since common implementation returns correct value

* [again] don't use "quiet" option for mvn tests...hard to figure out what's wrong when failed.

* normalize targets to 80 char strings for ATN serialization, except Java which needs big strings for efficiency.

* Update actions.md

fixed a small typo

* Rename `CodeGenerator.createCodeGenerator` to `CodeGenerator.create`

* Replace constants on string literals in `appendEscapedCodePoint`

* Restore API of Target

getLanguage(): protected -> public as it was before

appendUnicodeEscapedCodePoint(int codePoint, StringBuilder sb, boolean escape): protected -> private (it's a new helper method, no need for API now)

Added comment for appendUnicodeEscapedCodePoint

* Introduce caseInsensitive lexer rule option, fixes #3436

* don't ahead of time compile for DART. See 8ca8804ff1 (commitcomment-62642779)

* Simplify test rig related to timeouts (#3445)

* remove all -q quiet mvn options to see output on CI servers.

* run the various unit test classes in parallel rather than each individual test method, all except for Swift at the moment: `-Dparallel=classes -DthreadCount=4`

* use bigger machine at circleci

* No more test groups like parser1, parser2.

* simplify Swift like the other tests

* fix whitespace issues

* use 4.10 not 4.9.4

* improve releasing antlr doc

* Add Support For Swift Package Manager (#3132)

* Add Swift Package Manager Support

* Swift Package Dynamic

* 【fix】【test】Fix run process path

Co-authored-by: Terence Parr <parrt@cs.usfca.edu>

* use src 11 for tool, but 8 for plugin/runtime (#3450)

* use src 11 for tool, but 8 for plugin/runtime/runtime-tests.
* use 11 in CI builds

* cpp/cmake: Fix library install directories (#3447)

This installs DLLs in bin directory instead of lib.

* Python local import fixes (#3232)

* Fixed pygrun relative import issue

* Added name to contributors.txt

Co-authored-by: Terence Parr <parrt@cs.usfca.edu>

* Update javadoc to 8 and 11 (#3454)

* no need for plugin in runtime, always gen svg from dot for javadoc, gen 1.8 not 1.7 doc for runtime. Gen 11 for tool.

* tweak doc for 1.8 runtime.  Test rig should gen 1.8 not 1.7

* [Go] Fix (*BitSet).equals (#3455)

* set tool version for testing

* oops reversion tool version as it's not sync'd with runtime and not time to release yet.

* Remove unused variable from generated code (#3459)

* [C++] Fix bugs in UnbufferedCharStream (#3420)

* Escape bad words during grammar generation (#3451)

* Escape reserved words during grammar generation, fixes #1070 (for -> for_ but RULE_for)

Deprecate USE_OF_BAD_WORD

* Make name and escapedName consistent across tool and codegen classes

Fix other pull request notes

* Rename NamedActionChunk to SymbolRefChunk

* try out windows runners

* rename workflow

* Update windows.yml

Fix cmd line issue

* fix maven issue on windows

* use jdk 11

* remove arch arg

* display Github status for windows

* try testing python3 on windows

* try new run for python3 windows

* try new run for python3 windows (again)

* try new run for python3 windows (again2)

* try new run for python3 windows (again3)

* try new run for python3 windows (again4)

* try new run for python3 windows (again5)

* try new run for python3 windows

* try new run for python3 windows

* try new run for python3 windows

* ugh i give up. python won't install on github actions.

* Update windows.yml

try python 3

* Update windows.yml

* Update run-tests-python3.cmd

* Update run-tests-python3.cmd

* Create run-tests-python2.cmd

* Update windows.yml

* Update run-tests-python2.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Create run-tests-javascript.cmd

* Update run-tests-javascript.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Create run-tests-csharp.cmd

* Update windows.yml

* fix warnings in C# CI

* Update windows.yml

* Update windows.yml

* Create run-tests-dart.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update run-tests-dart.cmd

* Update run-tests-dart.cmd

* Update run-tests-dart.cmd

* Update run-tests-dart.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Create run-tests-go.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* GitHub action php (#3474)

* Update windows.yml

* Create run-tests-php.cmd

* Update run-tests-php.cmd

* Update run-tests-php.cmd

* Update run-tests-php.cmd

* Update run-tests-php.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update run-tests-php.cmd

* Update windows.yml

* Cleanup ci (#3476)

* Delete .appveyor directory

* Delete .travis directory

* Improve CI concurrency (#3477)

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Optimize toArray

replace toArray(new T[size]) with toArray(new T[0]) for better performance

https://shipilev.net/blog/2016/arrays-wisdom-ancients/#_conclusion

* add contributor

* resolve conflicts

* fix-maven-concurrency (#3479)

* fix-maven-concurrency

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update run-tests-python2.cmd

* Update run-tests-python3.cmd

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update windows.yml

* Update run-tests-php.cmd

* Update windows.yml

* Update run-tests-dart.cmd

* Update run-tests-csharp.cmd

* Update run-tests-go.cmd

* Update run-tests-java.cmd

* Update run-tests-javascript.cmd

* Update run-tests-php.cmd

* Update run-tests-python2.cmd

* Update run-tests-python3.cmd

* increase Windows CI concurrency for all targets except Dart

* Preserve line separators for input runtime tests data (#3483)

* Preserve line separators for input data in runtime tests, fix test data

Refactor and improve performance of BaseRuntimeTest

* Add LineSeparator (\n, \r\n) tests

* Set up .gitattributes for LineSeparator_LF.txt (eol=lf) and LineSeparator_CRLF.txt (eol=crlf)

* Restore `\n` for all input in runtime tests, add extra LexerExec tests (LineSeparatorLf, LineSeparatorCrLf)

* Add generated LargeLexer test, remove LargeLexer.txt descriptor

* tweak name to be GeneratedLexerDescriptors

* [JavaScript] Migrate from jest to jasmine

* [C++] Fix Windows min/max macro collision

* [C++] Update cmake README.md to C++17

* remove unnecessary comparisons.

* Add useful function writeSerializedATNIntegerHistogram for writing out information concerning how many of each integer value appear in a serialized ATN.

* fix  comment indicating what goes in the serialized ATN.

* move writeSerializedATNIntegerHistogram out of runtime.

* follow guidelines

* Fix .interp file parsing test for the Java runtime.

Also includes separating the generation of the .interp file from writing it out so that we can use both independently.

* Delete files no longer needed. Should have been part of https://github.com/antlr/antlr4/pull/3520

* [C++] Optimizations and cleanups and const correctness, oh my

* [C++] Optimize LL1Analyzer

* [C++] Fix missing virtual destructors

* Remove not used PROTECTED, PUBLIC, PRIVATE tokens from ANTLRLexer.g

* Remove ANTLR 3 stuff from ANTLR grammars, deprecate ANTLR 3 errors

* Remove not used imaginary tokens from ANTLRParser.g

* Fix misprints in grammars

* ATN serialized data: remove shifting by 2, remove UUID; fix #3515

Regenerate XPathLexer files

* Disable native runtime tests (see #3521)

* Implement Java-specific ATN data optimization (+-2 shift)

* [C++] Remove now unused antlrcpp::Guid

* pull new branch diagram from master

* use dev not master branch for CI github

* update doc from master

* add back missing author

* [C++] Fix const correctness in ATN and DFA

* keep getSerializedATNSegmentLimit at max int

* Fixes #3259 make InErrorRecoveryMode public for go

* Change code gen template to capitalize InErrorRecoveryMode

* [C++] Improve multithreaded performance, fix TSAN error, and fix profiling ATN simulator setup bug

* Get rid of unnecessary allocations and calculations in SerializedATN

* Get rid of excess char escaping in generated files, decrease size of output files

Fix creation of excess fragments for Dart, Cpp, PHP runtimes

* Swift: fix binary serialization and use instead of JSON

* Fix targetCharValueEscape, make them final and static

* [C++] Cleanup ATNDeserializer and remove related deprecated methods from ATNSimulator

* Fix for #3557 (getting "go test" to work again).

* Convert Python2/3 to use int arrays not strings for ATN encodings (#3561)

* Convert Python2/3 to use int arrays not strings for ATN encodings. Also make target indicate int vs string.

* rename and reverse ATNSerializedAsInts

* add override

* remove unneeded method

* [C++] Drastically improve multi-threaded performance (#3550)

Thanks guys. A major advancement.

* [C++] Remove duplicate includes and remove unused includes (#3563)

* [C++] Lazily deserialize ATN in generated code (#3562)

* [Docs] Update Swift Docs (#3458)

* Add Swift Package Manager Support

* Swift Package Dynamic

* 【fix】【test】Fix run process path

* [Docs] [Swift] update link, remove expired descriptions

Co-authored-by: Terence Parr <parrt@cs.usfca.edu>

* Ascii only ATN serialization (#3566)

* go back to generating pure ascii ATN serializations to avoid issues where target compilers might assume ascii vs utf-8.

* forgot I had to change php on previous ATN serialization tweak.

* change how we escapeChar() per target.

* oops; gotta use escapeChar method

* rm unneeded case

* add @override

* use ints not chars for C# (#3567)

* use ints not chars for C#

* oops. remove 'quotes'

* regen from XPathLexer.g4

* simplify ATN with bypass alts mechanism in Java.

* Change string to int[] for serialized ATN for C#; removed unneeded `use System` from XPathLexer.g4; regen that grammar.

* [C++] Use camel case name in generated lexers and parsers (#3565)

* Change string to int array for serialized ATN for JavaScript (#3568)

* perf: Add default implementation for Visit in ParseTreeVisitor.  (#3569)

* perf: Add default implementation for Visit in ParseTreeVisitor.

Reference: ad29539cd2/runtime/Java/src/org/antlr/v4/runtime/tree/AbstractParseTreeVisitor.java (L18)

* doc: add contributor

* Don't use utf decoding...these are just ints (#3573)

* [Go] Cleanup and fix ATN deserialization verification (#3574)

* [C++] Force generated static data type name to titlecase (#3572)

* Use int array not string for ATN in Swift (#3575)

* [C++] Fix generated Lexer static data constructor (#3576)

* Use int array not string for ATN in Dart (#3578)

* Fix PHP codegen to support int ATN serialization (#3579)

* Update listener documentation to satisfy the discussion about improving exception handling: https://github.com/antlr/antlr4/discussions/3162

* tweak

* [C++] Remove unused LexerATNSimulator::match_calls (#3570)

* [C++] Remove unused LexerATNSimulator::match_calls

* Remove match_calls from other targets

* [Java] Preserve serialized ATN version 3 compatibility (#3583)

* add jcking to the contributors list

* Update releasing-antlr.md

* [C++] Avoid using dynamic_cast where possible by using hand rolled RTTI (#3584)

* Revert "[Java] Preserve serialized ATN version 3 compatibility (#3583)"

This reverts commit 01bc811557.

* [C++] Add ANTLR4CPP_PUBLIC attributes to various symbols (#3588)

* Update editorconfig for c++ (#3586)

* Make it easier to contribute: Add c++ configuration for .editorconfig.

Using the observed style with 2 indentation spaces.

Signed-off-by: Henner Zeller <hzeller@google.com>

* Add hzeller to contributors.txt

Signed-off-by: Henner Zeller <hzeller@google.com>

* Fix code style and typing to support PHP 8 (#3582)

* [Go] Port locking algorithm from C++ to Go (#3571)

* Use linux DCO not our old contributors certificate of origin

* [C++] Fix bugs in SemanticContext (#3595)

* [Go] Do not export Array2DHashSet which is an implementation detail (#3597)

* Revert "Use linux DCO not our old contributors certificate of origin"

This reverts commit b0f8551c9a.

* Use signed ints for ATN serialization not uint16, except for java (#3591)

* refactor serialize so we don't need comments

* more cleanup during refactor

* store language in serializer obj

* A lexer rule token type should never be -1 (EOF). 0 is fragment but then must be > 0.

* Go uses int not uint16 for ATN now. java/go/python3 pass

* remove checks for 0xFFFF in Go.

* C++ uint16_t to int for ATN.

* add mac php dir; fix type on accept() for generated code to be mixed.

* Add test from @kvanTTT. This PR fixes https://github.com/antlr/antlr4/issues/3555 for non-Java targets.

* cleanup and add big lexer from https://github.com/antlr/antlr4/pull/3546

* increase mvn mem size to 2G

* increase mvn mem size to 8G

* turn off the big ATN lexer test as we have memory issues during testing.

* Fixes #3592

* Revert "C++ uint16_t to int for ATN."

This reverts commit 4d2ebbf567.

# Conflicts:
#	runtime/Cpp/runtime/src/atn/ATNSerializer.cpp
#	runtime/Cpp/runtime/src/tree/xpath/XPathLexer.cpp

* C++ uint16_t to int32_t for ATN.

* rm unnecessary include file, updating project file. get rid of the 0xFFFF does in the C++ deserialization

* rm refs to 0xFFFF in swift

* javascript tests were running as Node...added to ignore list.

* don't distinguish between 16 and 32 bit char sets in serialization; Python2/3  updated to work with this change.

* update C++ to deserialize only 32-bit sets

* 0xFFFF -> -1 for C++ target.

* get other targets to use 32-bit sets in serialization. tests pass locally.

* refactor to reduce code size

* add comment

* oops. comment out call to writeSerializedATNIntegerHistogram(). I wonder if this is why it ran out of memory during testing?

* all but Java, Node, PHP, Go work now for the huge lexer file; I have set them to ignore.  note that the swift target takes over a minute to lex it.  I've turned off Node but it does not seem to terminate but it could terminate eventually.

* all but Java, Node, PHP, Go work now for the huge lexer file; I have set them to ignore.  note that the swift target takes over a minute to lex it.  I've turned off Node but it does not seem to terminate but it could terminate eventually.

* Turn off this big lexer because we get memory errors during continuous integration

* Intermediate commit where I have shuffled around all of the -1 flipping and bumping by two.  work still needs to be done because the token stream rewriter stuff fails. and I assume the other decoding for human readability testing if doesn't work

* convert decode to use int[]; remove dead code. don't use serializeAsChar stuff. more tests pass.

* more tests passing. simplify. When copying atn, must run ATN through serializer to set some state flags.

* 0xFFFD+ are not valid char

* clean up. tests passing now

* huge clean up. Got Java working with 32-bit ATNs!Still working on cleanup but I want to run the tests

* Cleanup the hack I did earlier; everything still seems to work

* Use linux DCO not our old contributors certificate of origin

* remove bump-by-2 code

* clean up per @kvanTTT. Can't test locally on this box. Will see what CI says.

* tweak comment

* Revert "Use linux DCO not our old contributors certificate of origin"

This reverts commit b0f8551c9a.

* see if C++ works in CI for huge ATN

* Use linux DCO not our old contributors certificate of origin (#3598)

* Use linux DCO not our old contributors certificate of origin

* Revert "Use linux DCO not our old contributors certificate of origin"

This reverts commit b0f8551c9a.

* use linux DCO

* use linux DCO

* Use linux DCO not our old contributors certificate of origin

* update release documentation

Signed-off-by: Terence Parr <parrt@antlr.org>

* Equivalent of https://github.com/antlr/antlr4/pull/3537

* clean up setup

* clean up doc version

* [Swift] improvements to equality functions (#3302)

* fix default equality

* equality cases

* optional unwrapping

* [Swift] Use for in loops (#3303)

* common for in loops

* reversed loop

* drop first loop

* for in with default BitSet

* [Go] Fix symbol collision in generated lexers and parsers (#3603)

* [C++] Refactor and optimize SemanticContext (#3594)

* [C++] Devirtualize hand rolled RTTI for performance (#3609)

* [C++] Add T::is for type hierarchy checks and remove some dynamic_cast (#3612)

* [C++] Avoid copying statically generated serialized ATNs (#3613)

* [C++] Refactor PredictionContext and yet more performance improvements (#3608)

* [C++] Cleanup DFA, DFAState, LexerAction, and yet more performance improvements (#3615)

* fix dependabot issues

* [Swift] use stdlib (single pass) (#3602)

* this was added to the stdlib in Swift 5

* &>> is defined as lhs >> (rhs % lhs.bitwidth)

* the stdlib has these

* reduce loops

* use indices

* append(contentsOf:)

* Array literal init works for sets too!

* inline and remove bit query functions

* more optional handling (#3605)

* [C++] Minor improvements to PredictionContext (#3616)

* use php runtime dev branch to test dev

* update doc to be more explicit about the interaction between lexer actions and semantic predicates; Fixes #3611. Fixes #3606.

Signed-off-by: Terence Parr <parrt@antlr.org>

* Refactor js runtime in preparation of future improvements

* refactor, 1 file per class, use import, use module semantics, use webpack 5, use eslint

* all tests pass

* simplifications and alignment with standard js idioms

* simplifications and alignment with standard js idioms

* support reading legacy ATN

* support both module and non-module imports

* fix failing tests

* fix failing tests

* No longer necessary too generate sets or single atom transit that are bigger than 16bits. (#3620)

* Updated getting started with Cpp documentation. (#3628)

Included specific examples of using ANTLR4_TAG and ANTLR4_ZIP_REPOSITORY in the sample CMakeLists file.

* [C++] Free ATNConfig lookup set in readonly ATNConfigSet (#3630)

* [C++] Implement configurable PredictionContextMergeCache (#3627)

* Allow to choose to switch off building tests in C++ (#3624)

The new option to cmake ANTLR_BUILD_CPP_TESTS is default
on (so the behavior is as before), but it provides a way to
switch off if not needed.

The C++ tests pull in an external dependency (googletests),
which might conflict if ANTLR is used as a subproject in
another cmake project.

Signed-off-by: Henner Zeller <h.zeller@acm.org>

* Fix NPE for undefined label, fix antlr#2788

* An interval ought to be a value

Interval was a pointer to 2 Ints
it ought to be just 2 Ints, which is smaller and more semantically correct,
with no need for a cache.

However, this technically breaks metadata and AnyObject conformance but people shouldn't be relying on those for an Interval.

* [C++] Remove more dynamic_cast usage

* [C++] Introduce version macros

* add license prefix

* Prep 4.10 (#3599)

* Tweak doc

* Swift was referring to hardcoded version

* Start version update script.

* add files to update

* clean up setup

* clean up setup

* clean up setup

* don't need file

* don't need file

* Fixes #3600.  add instructions and associated code necessary to build the xpath lexers.

* clean up version nums

* php8

* php8

* php8

* php8

* php8

* php8

* php8

* php8

* tweak doc

* ok, i give up. php won't  bump up too v8

* tweak doc

* version number bumped to 4.10 in runtime.

* Change the doc for releasing and update to use latest ST 4.3.2

* fix dart version to 4.10.0

* cmd files Cannot use export bash command.

* try fixing php ci again

* working on deploy

Signed-off-by: Terence Parr <parrt@antlr.org>

* php8 always install.

* set js to 4.10.0 not 4.10

* turn off apt update for php circleci

* try w/o cimg/php

* try setting branch

* ok i give up

* tweak

* update docs for release.

* php8 circleci

* use 3.5.3 antlr

* use 3.5.3-SNAPSHOT antlr

* use full 3.5.3 antlr

* [Swift] reduce Optionals in APIs (#3621)

* ParserRuleContext.children

see comment in removeLastChild

* TokenStream.getText

* Parser._parseListeners

this might require changes to the code templates?

* ATN {various}

* make computeReachSet return empty, not nil

* overrides refine optionality

* BufferedTokenStream getHiddenTokensTo{Left, Right} return empty not nil

* Update Swift.stg

* avoid breakage by adding overload of `getText` in extension

* tweak to kick off build

Signed-off-by: Terence Parr <parrt@antlr.org>

* try     parallelism: 4 circleci

* Revert "[Swift] reduce Optionals in APIs (#3621)"

This reverts commit b5ccba03c8.

* tweaks to doc

* Improve the deploy  script and tweak the released doc.

* use 4.10 not Snapshot for scripts

Co-authored-by: Ivan Kochurkin <kvanttt@gmail.com>
Co-authored-by: Alexandr <60813335+Alex-Andrv@users.noreply.github.com>
Co-authored-by: 100mango <100mango@users.noreply.github.com>
Co-authored-by: Biswapriyo Nath <nathbappai@gmail.com>
Co-authored-by: Benjamin Spiegel <bspiegel11@gmail.com>
Co-authored-by: Justin King <jcking@google.com>
Co-authored-by: Eric Vergnaud <eric.vergnaud@wanadoo.fr>
Co-authored-by: Harry Chan <harry.chan@codersatlas.com>
Co-authored-by: Ken Domino <kenneth.domino@domemtech.com>
Co-authored-by: chenquan <chenquan.dev@gmail.com>
Co-authored-by: Marcos Passos <marcospassos@users.noreply.github.com>
Co-authored-by: Henner Zeller <h.zeller@acm.org>
Co-authored-by: Dante Broggi <34220985+Dante-Broggi@users.noreply.github.com>
Co-authored-by: chris-miner <94078897+chris-miner@users.noreply.github.com>
This commit is contained in:
Terence Parr 2022-04-10 13:41:31 -07:00 committed by GitHub
parent 97c793e446
commit e1455c06f0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1237 changed files with 30225 additions and 50066 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

128
.appveyor/workflow.yml
View File

@ -1,128 +0,0 @@
environment:
matrix:
- job_name: java-tool-and-runtime
- job_name: csharp-runtime
job_depends_on: java-tool-and-runtime
- job_name: dart-runtime
job_depends_on: java-tool-and-runtime
- job_name: go-runtime
job_depends_on: java-tool-and-runtime
- job_name: javascript-runtime
job_depends_on: java-tool-and-runtime
- job_name: php-runtime
job_depends_on: java-tool-and-runtime
- job_name: python2-runtime
job_depends_on: java-tool-and-runtime
- job_name: python3-runtime
job_depends_on: java-tool-and-runtime
matrix:
fast_finish: false
version: '4.9.3-SNAPSHOT+AppVeyor.{build}'
cache:
- '%USERPROFILE%\.m2'
- '%USERPROFILE%\.nuget\packages -> **\project.json'
image: Visual Studio 2019
# not using MSBuild
build: off
for:
- matrix:
only:
- job_name: java-tool-and-runtime
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd tool-testsuite
- mvn -q test
- cd ..\runtime-testsuite
- mvn -q -Dtest=java.* test
- matrix:
only:
- job_name: csharp-runtime
build_script:
- mvn -q -DskipTests install --batch-mode
- dotnet build runtime/CSharp/src/Antlr4.csproj -c Release
after_build:
- dotnet pack runtime/CSharp/src/Antlr4.csproj -c Release
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=csharp.* test
artifacts:
- path: 'runtime\**\*.nupkg'
name: NuGet
- matrix:
only:
- job_name: dart-runtime
install:
- cinst -y dart-sdk --version=2.12.1
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=dart.* test -Dantlr-dart-dart="C:\tools\dart-sdk\bin\dart.exe" -Dantlr-dart-pub="C:\tools\dart-sdk\bin\pub.bat" -Dantlr-dart-dart2native="C:\tools\dart-sdk\bin\dart2native.bat"
- matrix:
only:
- job_name: go-runtime
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=go.* test
- matrix:
only:
- job_name: javascript-runtime
install:
- cinst nodejs.install
- node --version
- npm --version
- npm install -g yarn@v1.22.10
build_script:
- cd runtime\JavaScript\
- npm install
- npm link
- cd ..\..
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime\JavaScript\
- yarn test
- cd ..\..
- cd runtime-testsuite
- mvn -q -Dtest=javascript.* test -Dantlr-javascript-npm="C:\Program Files\nodejs\npm.cmd" -Dantlr-javascript-nodejs="C:\Program Files\nodejs\node.exe"
- matrix:
only:
- job_name: php-runtime
install:
- git clone https://github.com/antlr/antlr-php-runtime.git
- mv antlr-php-runtime runtime/PHP
- cinst -y php --params "/InstallDir:C:\tools\php"
- cinst -y composer
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=php.* test -Dantlr-php-php="C:\tools\php\php.exe"
- matrix:
only:
- job_name: python2-runtime
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=python2.* test -Dantlr-python2-python="C:\Python27\python.exe"
- matrix:
only:
- job_name: python3-runtime
build_script:
- mvn -q -DskipTests install --batch-mode
test_script:
- cd runtime-testsuite
- mvn -q -Dtest=python3.* test -Dantlr-python3-python="C:\Python35\python.exe"

20
.circleci/config.yml
View File

@ -3,7 +3,7 @@ version: 2.1
jobs:
test_tool_and_runtime_java:
docker:
- image: cimg/openjdk:8.0
- image: cimg/openjdk:11.0
steps:
- checkout
- run:
@ -13,13 +13,13 @@ jobs:
name: test runtime
command: |
cd runtime-testsuite
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=java.* test
mvn -Dparallel=classes -DthreadCount=4 -Dtest=java.** test
cd ..
- run:
name: test tool
command: |
cd tool-testsuite
mvn -q -Dparallel=methods -DthreadCount=4 test
mvn -Dparallel=classes -DthreadCount=4 test
cd ..
test_runtime:
parameters:
@ -32,7 +32,11 @@ jobs:
type: string
default: java
docker:
- image: cimg/openjdk:8.0
- image: cimg/openjdk:11.0
environment:
MAVEN_OPTS: -Xmx512m
parallelism: 4
resource_class: large
environment:
TARGET: << parameters.target >>
GROUP: << parameters.test-group >>
@ -57,10 +61,4 @@ workflows:
- test_runtime:
matrix:
parameters:
target: [ dart, go, python2, python3, javascript, php ]
- test_runtime:
matrix:
parameters:
# target: [ cpp, dotnet, swift ]
target: [ cpp, dotnet, dart ]
test-group: [ LEXER, PARSER, RECURSION ]
target: [ dart, go, python2, python3, javascript, php, cpp, dotnet ]

4
.circleci/scripts/install-linux-javascript.sh
View File

@ -9,10 +9,6 @@ sudo apt-get install -y nodejs
echo node version: $(node --version)
echo "done installing nodejs"
echo "installing yarn..."
sudo npm install -g yarn@v1.22.10
echo "done installing yarn"
echo "packaging javascript runtime..."
pushd runtime/JavaScript
sudo npm install

14
.circleci/scripts/install-linux-php.sh
View File

@ -2,15 +2,19 @@
set -euo pipefail
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo apt-get update -qq
sudo apt install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo apt update
sudo apt install php-all-dev
sudo apt install wget php8.0-cli php8.0-zip unzip
wget -O composer-setup.php https://getcomposer.org/installer
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
sudo apt install php8.0
sudo apt install php8.0-mbstring
sudo apt install php8.0-xml
php -v
sudo apt install composer
git clone https://github.com/antlr/antlr-php-runtime.git runtime/PHP
composer install -d runtime/PHP

13
.circleci/scripts/run-tests-cpp.sh
View File

@ -3,15 +3,6 @@
set -euo pipefail
pushd runtime-testsuite
echo "running maven tests..."
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=cpp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=cpp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=cpp.* test
else
mvn -q -Dtest=cpp.* test
fi
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=cpp.** test
popd

4
.circleci/scripts/run-tests-dart.sh
View File

@ -6,6 +6,6 @@ dart --version
pushd runtime-testsuite
echo "running maven tests..."
# mvn -q -Dparallel=classes -DthreadCount=4 -Dtest=dart.* test
mvn -q -Dtest=dart.* test
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=dart.** test
popd

14
.circleci/scripts/run-tests-dotnet.sh
View File

@ -2,15 +2,7 @@
set -euo pipefail
pushd runtime-testsuite/
echo "running maven tests..."
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=csharp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=csharp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=csharp.* test
else
mvn -q -Dtest=csharp.* test
fi
pushd runtime-testsuite
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=csharp.** test
popd

5
.circleci/scripts/run-tests-go.sh
View File

@ -6,5 +6,6 @@ go version
pushd runtime-testsuite
echo "running maven tests..."
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=go.* test
popd
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=go.** test
popd

9
.circleci/scripts/run-tests-javascript.sh
View File

@ -6,8 +6,8 @@ declare -i RESULT=0
pushd runtime/JavaScript
echo "running jest tests..."
yarn test
echo "running jasmine tests..."
npm test
RESULT+=$?
popd
@ -15,9 +15,10 @@ popd
pushd runtime-testsuite
echo "running maven tests..."
mvn -q -Dtest=javascript.* test
export MAVEN_OPTS="-Xmx8g"
mvn -Dtest=javascript.** test
RESULT+=$?
popd
exit $RESULT
exit $RESULT

3
.circleci/scripts/run-tests-php.sh
View File

@ -7,5 +7,6 @@ php -v
php_path=$(which php)
pushd runtime-testsuite
echo "running maven tests..."
mvn -q -DPHP_PATH="${php_path}" -Dparallel=methods -DthreadCount=4 -Dtest=php.* test
export MAVEN_OPTS="-Xmx8g"
mvn -DPHP_PATH="${php_path}" -Dparallel=classes -DthreadCount=4 -Dtest=php.** test
popd

32
.circleci/scripts/run-tests-python2.sh
View File

@ -4,21 +4,19 @@ set -euo pipefail
python2 --version
pushd runtime/Python2/tests
echo "running native tests..."
python2 run.py
rc=$?
if [ $rc != 0 ]; then
echo "failed running native tests"
fi
popd
# TODO: https://github.com/antlr/antlr4/issues/3521
#
# pushd runtime/Python2/tests
# echo "running native tests..."
# python2 run.py
# rc=$?
# if [ $rc != 0 ]; then
# echo "failed running native tests"
# fi
# popd
if [ $rc == 0 ]; then
pushd runtime-testsuite
echo "running maven tests..."
mvn -q -Dtest=python2.* test
rc=$?
popd
fi
# return $rc
pushd runtime-testsuite
echo "running maven tests..."
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=python2.** test
popd

32
.circleci/scripts/run-tests-python3.sh
View File

@ -4,21 +4,19 @@ set -euo pipefail
python3 --version
pushd runtime/Python3/tests
echo "running native tests..."
python3 run.py
rc=$?
if [ $rc != 0 ]; then
echo "failed running native tests"
fi
# TODO: https://github.com/antlr/antlr4/issues/3521
#
# pushd runtime/Python3/tests
# echo "running native tests..."
# python3 run.py
# rc=$?
# if [ $rc != 0 ]; then
# echo "failed running native tests"
# fi
# popd
pushd runtime-testsuite
echo "running maven tests..."
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=python3.** test
popd
if [ $rc == 0 ]; then
pushd runtime-testsuite
echo "running maven tests..."
mvn -q -Dtest=python3.* test
rc=$?
popd
fi
# return $rc

39
.circleci/scripts/run-tests-swift.sh
View File

@ -1,27 +1,22 @@
#!/bin/bash
# Appears to be unused
set -euo pipefail
pushd runtime/Swift
echo "running native tests..."
./boot.py --test
rc=$?
if [ $rc != 0 ]; then
echo "failed running native tests"
fi
popd
# TODO: https://github.com/antlr/antlr4/issues/3521
#
# pushd runtime/Swift
# echo "running native tests..."
# ./boot.py --test
# rc=$?
# if [ $rc != 0 ]; then
# echo "failed running native tests"
# fi
# popd
if [ $rc == 0 ]; then
pushd runtime-testsuite
echo "running maven tests..."
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=swift.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=swift.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=swift.* test
else
mvn -q -Dtest=swift.* test
fi
popd
fi
pushd runtime-testsuite
echo "running maven tests..."
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=swift.** test
popd

20
.clang-format Normal file
View File

@ -0,0 +1,20 @@
---
# This section defines defaults for all languages. Currently we derive ANTLR style from LLVM.
BasedOnStyle: LLVM
# Only use clang-format for C++ for now.
DisableFormat: true
---
# This section configures C++ formatting.
Language: Cpp
DisableFormat: false
Standard: c++17
# Prevent clang-format from attempting to pick the alignment and always use right alignment.
DerivePointerAlignment: false
# ANTLR existing style is to right align pointers and references.
PointerAlignment: Right
ReferenceAlignment: Right
# Some of ANTLR existing code is longer than the default 80, so use 100 for now.
ColumnLimit: 100
# Historically ANTLR has used indentation within namespaces, so replicate it.
NamespaceIndentation: Inner

8
.editorconfig
View File

@ -8,3 +8,11 @@ charset = utf-8
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = tab
ij_java_else_on_new_line = true
[*.{h,cpp}]
charset = utf-8
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 2

2
.gitattributes vendored
View File

@ -1,2 +1,2 @@
# This rule applies to all files which don't match another line below
* text=auto
* text=auto

24
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@ -1,5 +1,23 @@
<!--
Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors must "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership.
Thank you for proposing a contribution to the ANTLR project!
Make sure you are already in the contributors.txt file or add a commit to this pull request with the appropriate change. Thanks!
-->
(Please make sure your PR is in a branch other than dev or master
and also make sure that you derive this branch from dev.)
As of 4.10, ANTLR uses the Linux Foundation's Developer
Certificate of Origin, DCO, version 1.1. See either
https://developercertificate.org/ or file
contributors-cert-of-origin.txt in the main directory.
Each commit requires a "signature", which is simple as
using `-s` (not `-S`) to the git commit command:
git commit -s -m 'This is my commit message'
Github's pull request process enforces the sig and gives
instructions on how to fix any commits that lack the sig.
See https://github.com/apps/dco for more info.
No signature is required in this file (unlike the
previous ANTLR contributor's certificate of origin.)
-->

0
.github/scripts/install-dotnet-on-osx.sh → .github/scripts-macosx/install-dotnet-on-osx.sh vendored
View File

0
.github/scripts/mac-action-runner-readme.txt → .github/scripts-macosx/mac-action-runner-readme.txt vendored
View File

8
.github/scripts-macosx/run-tests-cpp.sh vendored Executable file
View File

@ -0,0 +1,8 @@
#!/bin/bash
set -euo pipefail
pushd runtime-testsuite
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=cpp.** test
popd

18
.github/scripts-macosx/run-tests-dotnet.sh vendored Executable file
View File

@ -0,0 +1,18 @@
#!/bin/bash
# Appears not to be used at moment
set -euo pipefail
export PATH=$PATH:~/.dotnet
# we need to build the runtime before test run, since we used "--no-dependencies"
# when we call dotnet cli for restore and build, in order to speed up
dotnet build -c Release -f netstandard2.0 runtime/CSharp/Antlr4.csproj
# run tests
pushd runtime-testsuite/
export MAVEN_OPTS="-Xmx8g"
mvn -Dparallel=classes -DthreadCount=4 -Dtest=csharp.** test
popd

34
.github/scripts/run-tests-swift.sh → .github/scripts-macosx/run-tests-swift.sh vendored
View File

@ -27,24 +27,20 @@ swift --version
swift build --version
# run swift tests
pushd runtime/Swift
./boot.py --test
rc=$?
popd
if [ $rc == 0 ]; then
# run java tests
cd runtime-testsuite/
if [ $GROUP == "LEXER" ]; then
mvn -e -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest="swift.*" test
elif [ $GROUP == "PARSER" ]; then
mvn -e -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest="swift.*" test
elif [ $GROUP == "RECURSION" ]; then
mvn -e -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest="swift.*" test
else
mvn -e -q -Dtest=swift.* test
fi
rc=$?
cat target/surefire-reports/*.dumpstream || true
fi
# TODO: https://github.com/antlr/antlr4/issues/3521
# pushd runtime/Swift
# ./boot.py --test
# rc=$?
# popd
# run java tests
cd runtime-testsuite/
# mvn -e -Dparallel=classes -DthreadCount=4 -Dtest=swift.** test
# I don't know swift enough to make it parallel. revert to single threaded
export MAVEN_OPTS="-Xmx8g"
mvn -e -Dtest=swift.** test
rc=$?
cat target/surefire-reports/*.dumpstream || true
exit $rc

5
.github/scripts-windows/run-tests-csharp.cmd vendored Normal file
View File

@ -0,0 +1,5 @@
dotnet build runtime/CSharp/src/Antlr4.csproj -c Release
dotnet pack runtime/CSharp/src/Antlr4.csproj -c Release
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dtest=csharp.** test
cd ..

5
.github/scripts-windows/run-tests-dart.cmd vendored Normal file
View File

@ -0,0 +1,5 @@
C:\ProgramData\chocolatey\bin\choco.exe -y install dart-sdk
cd runtime-testsuite
mvn -Dtest=dart.** test -Dantlr-dart-dart="C:\tools\dart-sdk\bin\dart.exe" -Dantlr-dart-pub="C:\tools\dart-sdk\bin\pub.bat" -Dantlr-dart-dart2native="C:\tools\dart-sdk\bin\dart2native.bat"
cd ..

3
.github/scripts-windows/run-tests-go.cmd vendored Normal file
View File

@ -0,0 +1,3 @@
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dtest=go.** test
cd ..

3
.github/scripts-windows/run-tests-java.cmd vendored Executable file
View File

@ -0,0 +1,3 @@
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dtest=java.** test
cd ..

3
.github/scripts-windows/run-tests-javascript.cmd vendored Normal file
View File

@ -0,0 +1,3 @@
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dtest=javascript.** test
cd ..

8
.github/scripts-windows/run-tests-php.cmd vendored Normal file
View File

@ -0,0 +1,8 @@
REM C:\ProgramData\chocolatey\bin\choco.exe install php -y --package-parameters='"/DontAddToPath"'
git clone https://github.com/antlr/antlr-php-runtime.git
move antlr-php-runtime runtime\PHP
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dtest=php.** test -Dantlr-php-php="C:\tools\php81\php.exe"
cd ..

3
.github/scripts-windows/run-tests-python2.cmd vendored Normal file
View File

@ -0,0 +1,3 @@
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dantlr-python2-python="C:\Python27\python.exe" -Dtest=python2.** test
cd ..

3
.github/scripts-windows/run-tests-python3.cmd vendored Normal file
View File

@ -0,0 +1,3 @@
cd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=2 -Dantlr-python3-python="C:\Python310\python.exe" -Dtest=python3.** test
cd ..

15
.github/scripts/run-tests-cpp.sh vendored
View File

@ -1,15 +0,0 @@
#!/bin/bash
set -euo pipefail
cd runtime-testsuite/
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=cpp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=cpp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=cpp.* test
else
mvn -q -Dtest=cpp.* test
fi

23
.github/scripts/run-tests-dotnet.sh vendored
View File

@ -1,23 +0,0 @@
#!/bin/bash
set -euo pipefail
export PATH=$PATH:~/.dotnet
# we need to build the runtime before test run, since we used "--no-dependencies"
# when we call dotnet cli for restore and build, in order to speed up
dotnet build -c Release -f netstandard2.0 runtime/CSharp/Antlr4.csproj
# run tests
cd runtime-testsuite/
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=csharp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=csharp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=csharp.* test
else
mvn -q -Dtest=csharp.* test
fi

15
.github/workflows/macosx.yml vendored
View File

@ -2,9 +2,9 @@ name: MacOSX
on:
push:
branches: [ master ]
branches: [ dev ]
pull_request:
branches: [ master ]
branches: [ dev ]
jobs:
build:
@ -12,23 +12,20 @@ jobs:
strategy:
fail-fast: false
matrix:
# TARGET: [swift, cpp, dotnet] disabling dotnet which is unstable on M1
TARGET: [swift, cpp]
GROUP: [LEXER, PARSER, RECURSION]
steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
- name: Set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 1.8
java-version: 11
- name: Set up Maven
uses: stCarolas/setup-maven@v4
with:
maven-version: 3.5.4
maven-version: 3.8.4
- name: Build tool with Maven
run: mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V
- name: Test with Maven
run: arch -x86_64 .github/scripts/run-tests-${{ matrix.TARGET }}.sh
run: .github/scripts-macosx/run-tests-${{ matrix.TARGET }}.sh
env:
TARGET: ${{ matrix.TARGET }}
GROUP: ${{ matrix.GROUP }}

77
.github/workflows/windows.yml vendored Normal file
View File

@ -0,0 +1,77 @@
name: Windows
on:
push:
branches: [ dev ]
pull_request:
branches: [ dev ]
jobs:
build:
runs-on: [self-hosted, windows, x64]
strategy:
fail-fast: false
matrix:
TARGET: [java, python2, python3, javascript, csharp, dart, go, php]
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Set up Maven
uses: stCarolas/setup-maven@v4
with:
maven-version: 3.5.4
# fails due to permissions, use global install
# - name: Set up Python 2
# if: matrix.TARGET == 'python2'
# uses: actions/setup-python@v2
# with:
# python-version: '2.x'
# architecture: 'x64'
# fails due to permissions, use global install
# - name: Set up Python 3
# if: matrix.TARGET == 'python3'
# uses: actions/setup-python@v2
# with:
# python-version: '3.x'
# architecture: 'x64'
- name: Set up Node 14
if: matrix.TARGET == 'javascript'
uses: actions/setup-node@v2
with:
node-version: '14'
- name: Setup Dotnet 3.1 and 5.0
if: matrix.TARGET == 'csharp'
uses: actions/setup-dotnet@v1
with:
dotnet-version: |
3.1.x
5.0.x
# fails due to os (Linux only), use global install
# - name: Setup Dart 2.12.1
# uses: dart-lang/setup-dart@v1
# with:
# sdk: 2.12.1
- name: Setup Go 1.13.1
if: matrix.TARGET == 'go'
uses: actions/setup-go@v2
with:
go-version: '^1.13.1'
# requires manually setting up pwsh
# fails due to incorrect script (missing printf)
# - name: Setup PHP 8.2
# if: matrix.TARGET == 'php'
# uses: shivammathur/setup-php@v2
# with:
# php-version: '8.2'
# extensions: mbstring
# tools: composer
- name: Build tool with Maven
run: mvn install -DskipTests=true -Darguments="-Dmaven.javadoc.skip=true" -B -V
- name: Test with Maven
run: .github/scripts-windows/run-tests-${{ matrix.TARGET }}.cmd
env:
TARGET: ${{ matrix.TARGET }}

7
.travis/before-install-linux-dart.sh
View File

@ -1,7 +0,0 @@
#!/bin/bash
set -euo pipefail
wget https://storage.googleapis.com/dart-archive/channels/stable/release/2.12.1/linux_packages/dart_2.12.1-1_amd64.deb
sudo dpkg -i ./dart_2.12.1-1_amd64.deb
sudo rm ./dart_2.12.1-1_amd64.deb
sudo apt-get install -f

10
.travis/before-install-linux-dotnet.sh
View File

@ -1,10 +0,0 @@
#!/bin/bash
set -euo pipefail
wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update; \
sudo apt-get install -y apt-transport-https && \
sudo apt-get update && \
sudo apt-get install -y dotnet-sdk-3.1

6
.travis/before-install-linux-go.sh
View File

@ -1,6 +0,0 @@
#!/bin/bash
set -euo pipefail
eval "$(sudo gimme 1.7.3)"
( go version ; go env ) || true

7
.travis/before-install-linux-javascript.sh
View File

@ -1,7 +0,0 @@
#!/bin/bash
set -euo pipefail
# use v14 and check
echo node version: $(node --version)

11
.travis/before-install-linux-php.sh
View File

@ -1,11 +0,0 @@
#!/bin/bash
set -euo pipefail
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo apt-get update -qq
php -v
git clone https://github.com/antlr/antlr-php-runtime.git
mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V

13
.travis/before-install-linux-swift.sh
View File

@ -1,13 +0,0 @@
set -euo pipefail
# install dependencies
# some packages below will be update, swift assumes newer versions
# of, for example, sqlite3 and libicu, without the update some
# tools will not work
sudo apt-get update
sudo apt-get install clang-3.6 libxml2
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.6 100
# This would fix a know linker issue mentioned in:
# https://bugs.swift.org/browse/SR-2299
sudo ln -sf ld.gold /usr/bin/ld

1111
.travis/before-install-osx-dotnet.sh
View File

File diff suppressed because it is too large Load Diff

14
.travis/run-tests-cpp.sh
View File

@ -1,14 +0,0 @@
#!/bin/bash
set -euo pipefail
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=cpp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=cpp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=cpp.* test
else
mvn -q -Dtest=cpp.* test
fi

4
.travis/run-tests-dart.sh
View File

@ -1,4 +0,0 @@
#!/bin/bash
set -euo pipefail
mvn -q -Dparallel=classes -DthreadCount=4 -Dtest=dart.* test

22
.travis/run-tests-dotnet.sh
View File

@ -1,22 +0,0 @@
#!/bin/bash
set -euo pipefail
export PATH=$PATH:/Users/travis/.dotnet
# we need to build the runtime before test run, since we used "--no-dependencies"
# when we call dotnet cli for restore and build, in order to speed up
dotnet build -c Release -f netstandard2.0 ../runtime/CSharp/src/Antlr4.csproj
# call test
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dparallel=classes -DthreadCount=4 -Dtest=csharp.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dparallel=classes -DthreadCount=4 -Dtest=csharp.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dparallel=classes -DthreadCount=4 -Dtest=csharp.* test
else
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=csharp.* test
fi

5
.travis/run-tests-go.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
set -euo pipefail
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=go.* test

7
.travis/run-tests-java.sh
View File

@ -1,7 +0,0 @@
#!/bin/bash
set -euo pipefail
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=java.* test
cd ../tool-testsuite
mvn test

9
.travis/run-tests-javascript.sh
View File

@ -1,9 +0,0 @@
#!/bin/bash
set -euo pipefail
cd ../runtime/JavaScript
npm install
npm link
cd ../../runtime-testsuite
mvn -q -Dparallel=methods -DthreadCount=1 -Dtest=javascript.* test

9
.travis/run-tests-php.sh
View File

@ -1,9 +0,0 @@
#!/bin/bash
set -euo pipefail
php_path=$(which php)
composer install -d ../runtime/PHP
mvn -q -DPHP_PATH="${php_path}" -Dparallel=methods -DthreadCount=4 -Dtest=php.* test

11
.travis/run-tests-python2.sh
View File

@ -1,11 +0,0 @@
#!/bin/bash
set -euo pipefail
python --version
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=python2.* test
cd ../runtime/Python2/tests
python run.py

11
.travis/run-tests-python3.sh
View File

@ -1,11 +0,0 @@
#!/bin/bash
set -euo pipefail
python3 --version
mvn -q -Dparallel=methods -DthreadCount=4 -Dtest=python3.* test
cd ../runtime/Python3/test
python3 run.py

41
.travis/run-tests-swift.sh
View File

@ -1,41 +0,0 @@
#!/bin/bash
set -euo pipefail
# linux specific setup, those setup have to be
# here since environment variables doesn't pass
# across scripts
if [ $TRAVIS_OS_NAME == "linux" ]; then
export SWIFT_VERSION=swift-5.0.1
export SWIFT_HOME=$(pwd)/swift/$SWIFT_VERSION-RELEASE-ubuntu16.04/usr/bin/
export PATH=$SWIFT_HOME:$PATH
# download swift
mkdir swift
curl https://swift.org/builds/$SWIFT_VERSION-release/ubuntu1604/$SWIFT_VERSION-RELEASE/$SWIFT_VERSION-RELEASE-ubuntu16.04.tar.gz -s | tar xz -C swift &> /dev/null
fi
if [ -z "${JAVA_HOME-}" ]
then
export JAVA_HOME="$(java -XshowSettings:properties -version 2>&1 |
grep 'java\.home' | awk '{ print $3 }')"
echo "export JAVA_HOME=$JAVA_HOME"
fi
# check swift
swift --version
swift build --version
pushd ../runtime/Swift
./boot.py --test
popd
if [ $GROUP == "LEXER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LexerTests" -Dtest=swift.* test
elif [ $GROUP == "PARSER" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.ParserTests" -Dtest=swift.* test
elif [ $GROUP == "RECURSION" ]; then
mvn -q -Dgroups="org.antlr.v4.test.runtime.category.LeftRecursionTests" -Dtest=swift.* test
else
mvn -q -Dtest=swift.* test
fi

20
CONTRIBUTING.md
View File

@ -2,11 +2,21 @@
1. [Fork](https://help.github.com/articles/fork-a-repo) the [antlr/antlr4 repo](https://github.com/antlr/antlr4), which will give you both key branches, `master` and `dev`
2. Make sure to `git checkout dev` in your fork so that you are working from the latest development branch
2. Install and configure [EditorConfig](http://editorconfig.org/) so your text editor or IDE uses the ANTLR 4 coding style
3. [Build ANTLR 4](doc/building-antlr.md)
4. [Run the ANTLR project unit tests](doc/antlr-project-testing.md)
5. Create a [pull request](https://help.github.com/articles/using-pull-requests/) with your changes and make sure you're comparing the `dev` branch in your fork to the `dev` branch from the `antlr/antlr4` repo:
3. Create and work from a branch from `dev` such as `git checkout -b your-branch-name`
4. Install and configure [EditorConfig](http://editorconfig.org/) so your text editor or IDE uses the ANTLR 4 coding style
5. [Build ANTLR 4](doc/building-antlr.md)
6. [Run the ANTLR project unit tests](doc/antlr-project-testing.md)
7. Create a [pull request](https://help.github.com/articles/using-pull-requests/) with your changes and make sure you're comparing your `dev`-derived branch in your fork to the `dev` branch from the `antlr/antlr4` repo:
<img src="doc/images/PR-on-dev.png" width="600">
**Note:** You must sign the `contributors.txt` certificate of origin with your pull request if you've not done so before.
**Note:** Each commit requires a "signature", which is simple as using `-s` (not
`-S`) to the git commit command:
```
git commit -s -m 'This is my commit message'
```
Github's pull request process enforces the sig and gives instructions on how to
fix any commits that lack the sig. See [Github DCO app](https://github.com/apps/dco)
for more info.

12
runtime/Swift/Package.swift → Package.swift
View File

@ -1,7 +1,5 @@
// swift-tools-version:4.0
// Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
// Use of this file is governed by the BSD 3-clause license that
// can be found in the LICENSE.txt file in the project root.
// swift-tools-version:5.3
import PackageDescription
@ -16,9 +14,11 @@ let package = Package(
targets: [
.target(
name: "Antlr4",
dependencies: []),
dependencies: [],
path: "./runtime/Swift/Sources/Antlr4"),
.testTarget(
name: "Antlr4Tests",
dependencies: ["Antlr4"]),
dependencies: ["Antlr4"],
path:"./runtime/Swift/Tests/Antlr4Tests"),
]
)

2
README.md
View File

@ -25,8 +25,6 @@ Targets such as Go that pull directly from the repository can use the default `m
$ go get github.com/antlr/antlr4/runtime/Go/antlr@dev
```
## Authors and major contributors
* [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu

56
antlr4-maven-plugin/pom.xml
View File

@ -8,23 +8,18 @@
<parent>
<groupId>org.antlr</groupId>
<artifactId>antlr4-master</artifactId>
<version>4.9.4-SNAPSHOT</version>
<version>4.10-SNAPSHOT</version>
</parent>
<artifactId>antlr4-maven-plugin</artifactId>
<packaging>maven-plugin</packaging>
<name>ANTLR 4 Maven plugin</name>
<description>Maven plugin for ANTLR 4 grammars</description>
<prerequisites>
<maven>3.0</maven>
</prerequisites>
<!-- Ancilliary information for completeness -->
<inceptionYear>2009</inceptionYear>
<properties>
<mavenVersion>3.3.9</mavenVersion>
<takariLifecycleVersion>1.11.12</takariLifecycleVersion>
<mavenVersion>3.8.4</mavenVersion>
</properties>
<!-- ============================================================================= -->
@ -39,13 +34,13 @@
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-plugin-api</artifactId>
<version>3.0.5</version>
<scope>compile</scope>
<version>3.8.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.codehaus.plexus</groupId>
<artifactId>plexus-compiler-api</artifactId>
<version>2.2</version>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.sonatype.plexus</groupId>
@ -64,26 +59,26 @@
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.maven.plugin-tools</groupId>
<artifactId>maven-plugin-annotations</artifactId>
<version>3.2</version>
<version>3.6.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.takari.maven.plugins</groupId>
<artifactId>takari-plugin-testing</artifactId>
<version>2.9.0</version>
<version>3.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-core</artifactId>
<version>${mavenVersion}</version>
<scope>test</scope>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
@ -94,14 +89,19 @@
<dependency>
<groupId>org.codehaus.plexus</groupId>
<artifactId>plexus-utils</artifactId>
<version>3.0.15</version>
<version>3.4.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-project</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.32</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.32</version>
</dependency>
</dependencies>
<build>
@ -120,7 +120,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-plugin-plugin</artifactId>
<version>3.6.0</version>
<version>3.6.2</version>
<configuration>
<!-- see http://jira.codehaus.org/browse/MNG-5346 -->
<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
@ -143,7 +143,6 @@
<plugin>
<groupId>io.takari.maven.plugins</groupId>
<artifactId>takari-lifecycle-plugin</artifactId>
<version>${takariLifecycleVersion}</version>
<extensions>true</extensions>
<executions>
<execution>
@ -155,6 +154,13 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<release>8</release>
</configuration>
</plugin>
</plugins>
</build>
@ -168,7 +174,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.4</version>
<version>3.3.1</version>
<configuration>
<quiet>true</quiet>
</configuration>
@ -176,8 +182,8 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jxr-plugin</artifactId>
<version>2.5</version>
<version>3.1.1</version>
</plugin>
</plugins>
</reporting>
</project>
</project>

2
antlr4-maven-plugin/src/main/java/org/antlr/mojo/antlr4/Antlr4Mojo.java
View File

@ -276,7 +276,7 @@ public class Antlr4Mojo extends AbstractMojo {
for (List<String> args : argumentSets) {
try {
// Create an instance of the ANTLR 4 build tool
tool = new CustomTool(args.toArray(new String[args.size()]));
tool = new CustomTool(args.toArray(new String[0]));
} catch (Exception e) {
log.error("The attempt to create the ANTLR 4 build tool failed, see exception report for details", e);
throw new MojoFailureException("Error creating an instanceof the ANTLR tool.", e);

52
developer-cert-of-origin.txt Normal file
View File

@ -0,0 +1,52 @@
As of 4.10, ANTLR uses the Linux Foundation's Developer
Certificate of Origin, DCO, version 1.1. See either
https://developercertificate.org/ or the text below.
Each commit requires a "signature", which is simple as
using `-s` (not `-S`) to the git commit command:
git commit -s -m 'This is my commit message'
Github's pull request process enforces the sig and gives
instructions on how to fix any commits that lack the sig.
See https://github.com/apps/dco for more info.
No signature is required in this file (unlike the
previous ANTLR contributor's certificate of origin.)
----- https://developercertificate.org/ ------
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

2
doc/actions.md
View File

@ -81,7 +81,7 @@ returnStat : 'return' expr {System.out.println("matched "+$expr.text);} ;
Using a rule label looks like this:
```
returnStat : 'return' e=expr {System.out.println("matched "+e.text);} ;
returnStat : 'return' e=expr {System.out.println("matched "+$e.text);} ;
```
You can also use `$` followed by the name of the attribute to access the value associated with the currently executing rule. For example, `$start` is the starting token of the current rule.

166
doc/antlr-project-testing.md
View File

@ -6,37 +6,41 @@ Because ANTLR supports multiple target languages, the unit tests are broken into
The runtime tests must be specified in a generic fashion to work across language targets. Furthermore, we must test the various targets from Java. This usually means Java launching processes to compile, say, C++ and run parsers.
As of 4.6, we use [a Java descriptor object](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) to describe each runtime test. Unit tests are grouped together into categories such as [ParserExecDescriptors](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java), which has multiple nested descriptor objects, one per test. For example, here is the start of that file:
As of 4.10, we use a Java descriptor file held as an [UniversalRuntimeTestDescriptor.java object](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/UniversalRuntimeTestDescriptor.java) to represent each runtime test. Each test is described with a text file with various sections and resides in a group directory; see [directories under descriptors dir](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors). Here is a sample test descriptor:
```java
public class ParserExecDescriptors {
public static class APlus extends BaseParserTestDescriptor {
public String input = "a b c";
public String output = "abc\n";
public String errors = "";
public String startRule = "a";
public String grammarName = "T";
```
[notes]
This is a regression test for blah blah blah...
/**
grammar T;
a : ID+ {
<writeln("$text")>
};
ID : 'a'..'z'+;
WS : (' '|'\n') -> skip;
*/
@CommentHasStringValue
public String grammar;
}
[type]
Parser
[grammar]
grammar T;
a : ID* {
<writeln("$text")>
};
ID : 'a'..'z'+;
WS : (' '|'\n') -> skip;
[start]
a
[input]
a b c
[output]
"""abc
"""
```
The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allows multi-line strings in Java. This kung fu is required so that we can use Java classes rather than StringTemplate group files to specify runtime tests (the legacy system used those and it was hard to get them right). Here are all the [Runtime test descriptors](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) organized into groups.
The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). Use triple-quotes `"""` when whitespace matters (usually input/output sections).
## Requirements
*out of date, at least for mono*
In order to perform the tests on all target languages, you need to have the following languages installed:
* `mono` (e.g., `brew install mono`) on non-Windows boxes (on Windows it uses the Microsoft .net stack). Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
@ -44,14 +48,14 @@ In order to perform the tests on all target languages, you need to have the foll
* Python 2.7
* Python 3.6
* Go
* Swift 4 (via XCode 10.x) tested currently only osx
* Swift (via XCode) tested currently only osx
* clang (for C++ target)
*
To **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests # make sure all artifacts are visible on this machine
```
Now, make sure C# runtime is built and installed locally.
@ -76,51 +80,6 @@ And the result of testing the entire subdirectory:
<img src=images/python3-tests.png width=400>
From `mvn`, on the commandline, you will see:
```bash
$ cd antlr4
$ mvn test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:53 min
[INFO] Finished at: 2016-11-16T15:36:56-08:00
[INFO] Final Memory: 44M/458M
[INFO] ------------------------------------------------------------------------
```
Note: That is actually result of running the much faster:
```bash
mvn -Dparallel=methods -DthreadCount=4 install
```
## Running test subsets
*From the `runtime-testsuite` dir*
@ -157,7 +116,7 @@ Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.* test
$ mvn -Dtest=java.** test
...
```
@ -190,12 +149,12 @@ Use this to run tests in parallel:
```bash
$ export MAVEN_OPTS="-Xmx1G"
$ mvn -Dparallel=methods -DthreadCount=4 test
$ mvn -Dparallel=classes -DthreadCount=4 test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
...
```
@ -203,55 +162,42 @@ This can be combined with other `-D` above.
## Adding a runtime test
To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
To add a new runtime test, first determine which [group (dir) of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/descriptors/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new descriptor file implementation by filling in one of these (omitting unused sections):
* [BaseParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseParserTestDescriptor.java); see example [APlus](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java#L7).
* [BaseDiagnosticParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseDiagnosticParserTestDescriptor) if you want to test parser diagnostic output; see [example output](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/FullContextParsingDescriptors.java#L16).
* [BaseCompositeParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeParserTestDescriptor.java); see example [BringInLiteralsFromDelegate](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java#L11)
* [BaseLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseLexerTestDescriptor.java); see example [ActionPlacement](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/LexerExecDescriptors.java#L12).
* [BaseCompositeLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeLexerTestDescriptor.java); see example [LexerDelegatorInvokesDelegateRule](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeLexersDescriptors.java#L11)
```
[notes]
[type]
[grammar]
Each descriptor object describes the following mandatory elements for the test:
[slaveGrammar]
* the test type
* the grammar
* the start rule
* the input text to parse or lex
* the expected output
* the expected errors
[start]
Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor object, creating a new nested class within the test group class. Modify the field definitions to suit your new problem.
[input]
If you need to create a whole new group of tests, it requires a new descriptor class; call it `XDescriptors`. Then, in each [target subdirectory](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime), you need to create a new test rig `TestX.java` file:
[output]
```java
package org.antlr.v4.test.runtime.java;
[errors]
import org.antlr.v4.test.runtime.BaseRuntimeTest;
import org.antlr.v4.test.runtime.RuntimeTestDescriptor;
import org.antlr.v4.test.runtime.descriptors.ListenersDescriptors;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
[flags]
@RunWith(Parameterized.class)
public class TestX extends BaseRuntimeTest {
public TestX(RuntimeTestDescriptor descriptor) {
super(descriptor,new Base<TARGET>Test());
}
@Parameterized.Parameters(name="{0}")
public static RuntimeTestDescriptor[] getAllTestDescriptors() {
return BaseRuntimeTest.getRuntimeTestDescriptors(XDescriptors.class, "<TARGET>");
}
}
[skip]
```
where `<TARGET>` is replaced with Java, Cpp, CSharp, Python2, ... in the various subdirectories.
Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor file, creating a new file within the test group dir. Modify the sections to suit your new problem.
### Ignoring tests
In order to turn off a test for a particular target, we need to use the `ignore` method. Given a target name, a descriptor object can decide whether to ignore the test. This is not always convenient but it is fully general and works well for the one case we have now where we have to ignore `Visitor` tests in all targets except JavaScript.
In order to turn off a test for a particular target, we need to use the `skip` section in the descriptor file. For example, the following skips PHP and Dart targets:
```
[skip]
PHP
Dart
```
### Target API/library testing

46
doc/building-antlr.md
View File

@ -7,7 +7,7 @@ Most programmers do not need the information on this page because they will simp
I will assume that the root directory is `/tmp` for the purposes of explaining how to build ANTLR in this document.
*As of 4.6, ANTLR tool and Java-target runtime requires Java 7.*
*As of 4.6, ANTLR tool and Java-target runtime requires Java 7. As of 4.10, we have verified that the tool itself builds with Java 8 and 11.*
# Get the source
@ -43,41 +43,35 @@ $ if [[ "$?" != "0" ]]; then sudo apt install -y maven; fi
# Compile
```bash
$ cd /tmp/antlr4
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn clean # must be separate, not part of install/compile
$ mvn -DskipTests install
The current maven build seems complicated to me because there is a dependency of the project on itself. The runtime tests naturally depend on the current version being available but it won't compile without the current version. Once you have the generated/installed jar, mvn builds but otherwise there's a dependency on what you are going to build. You will get this error when you try to clean but you can ignore it:
```
[INFO] ANTLR 4 Runtime Tests (3rd generation) ............. FAILURE [ 0.073 s]
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.287 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 4.915 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.315 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 2.393 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.078 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.019 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [ 1.986 s]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 0.513 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.005 s
[INFO] Finished at: 2016-11-21T11:42:42-08:00
[INFO] Final Memory: 52M/434M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.antlr:antlr4-maven-plugin:4.10-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.antlr:antlr4-maven-plugin:jar:4.10-SNAPSHOT -> [Help 1]
```
To be super squeaky clean, you can wipe out the repository cache, then do the build:
```
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
cd /tmp/antlr4 # or wherever you have the software
rm -rf ~/.m2/repository/org/antlr*
mvn clean
mvn -DskipTests install
```
**NOTE:** We do `install` not `compile` as tool tests and such refer to modules that must be pulled from the maven install local cache.
Once you have completed this process once and there is a jar hanging around in the repository cache.
# Installing libs to mvn cache locally
To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
$ mvn install -DskipTests # make sure all artifacts are visible on this machine
```
You should see these jars (when building 4.6-SNAPSHOT):
@ -85,8 +79,6 @@ You should see these jars (when building 4.6-SNAPSHOT):
```bash
/Users/parrt/.m2/repository/org/antlr $ find antlr4* -name '*.jar'
antlr4-maven-plugin/4.6-SNAPSHOT/antlr4-maven-plugin-4.6-SNAPSHOT.jar
antlr4-runtime-test-annotation-processors/4.6-SNAPSHOT/antlr4-runtime-test-annotation-processors-4.6-SNAPSHOT.jar
antlr4-runtime-test-annotations/4.6-SNAPSHOT/antlr4-runtime-test-annotations-4.6-SNAPSHOT.jar
antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT-tests.jar
antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT.jar
antlr4-runtime/4.6-SNAPSHOT/antlr4-runtime-4.6-SNAPSHOT.jar

80
doc/case-insensitive-lexing.md
View File

@ -1,80 +0,0 @@
# Case-Insensitive Lexing
In some languages, keywords are case insensitive meaning that `BeGiN` means the same thing as `begin` or `BEGIN`. ANTLR has two mechanisms to support building grammars for such languages:
1. Build lexical rules that match either upper or lower case.
* **Advantage**: no changes required to ANTLR, makes it clear in the grammar that the language in this case insensitive.
* **Disadvantage**: might have a small efficiency cost and grammar is a more verbose and more of a hassle to write.
2. Build lexical rules that match keywords in all uppercase and then parse with a custom [character stream](https://github.com/antlr/antlr4/blob/master/runtime/Java/src/org/antlr/v4/runtime/CharStream.java) that converts all characters to uppercase before sending them to the lexer (via the `LA()` method). Care must be taken not to convert all characters in the stream to uppercase because characters within strings and comments should be unaffected. All we really want is to trick the lexer into thinking the input is all uppercase.
* **Advantage**: Could have a speed advantage depending on implementation, no change required to the grammar.
* **Disadvantage**: Requires that the case-insensitive stream and grammar are used in correctly in conjunction with each other, makes all characters appear as uppercase/lowercase to the lexer but some grammars are case sensitive outside of keywords, errors new case insensitive streams and language output targets (java, C#, C++, ...).
For the 4.7.1 release, we discussed both approaches in [detail](https://github.com/antlr/antlr4/pull/2046) and even possibly altering the ANTLR metalanguage to directly support case-insensitive lexing. We discussed including the case insensitive streams into the runtime but not all would be immediately supported. I decided to simply make documentation that clearly states how to handle this and include the appropriate snippets that people can cut-and-paste into their grammars.
## Case-insensitive grammars
As a prime example of a grammar that specifically describes case insensitive keywords, see the
[SQLite grammar](https://github.com/antlr/grammars-v4/blob/master/sqlite/SQLite.g4). To match a case insensitive keyword, there are rules such as
```
K_UPDATE : U P D A T E;
```
that will match `UpdaTE` and `upDATE` etc... as the `update` keyword. This rule makes use of some generically useful fragment rules that you can cut-and-paste into your grammars:
```
fragment A : [aA]; // match either an 'a' or 'A'
fragment B : [bB];
fragment C : [cC];
fragment D : [dD];
fragment E : [eE];
fragment F : [fF];
fragment G : [gG];
fragment H : [hH];
fragment I : [iI];
fragment J : [jJ];
fragment K : [kK];
fragment L : [lL];
fragment M : [mM];
fragment N : [nN];
fragment O : [oO];
fragment P : [pP];
fragment Q : [qQ];
fragment R : [rR];
fragment S : [sS];
fragment T : [tT];
fragment U : [uU];
fragment V : [vV];
fragment W : [wW];
fragment X : [xX];
fragment Y : [yY];
fragment Z : [zZ];
```
No special streams are required to use this mechanism for case insensitivity.
## Custom character streams approach
The other approach is to use lexical rules that match either all uppercase or all lowercase, such as:
```
K_UPDATE : 'UPDATE';
```
Then, when creating the character stream to parse from, we need a custom class that overrides methods used by the lexer. Below you will find custom character streams for a number of the targets that you can copy into your projects, but here is how to use the streams in Java as an example:
```java
CharStream s = CharStreams.fromPath(Paths.get('test.sql'));
CaseChangingCharStream upper = new CaseChangingCharStream(s, true);
Lexer lexer = new SomeSQLLexer(upper);
```
Here are implementations of `CaseChangingCharStream` in various target languages:
* [C#](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingCharStream.cs)
* [Dart](https://github.com/antlr/antlr4/blob/master/doc/resources/case_changing_char_stream.dart)
* [Go](https://github.com/antlr/antlr4/blob/master/doc/resources/case_changing_stream.go)
* [Java](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingCharStream.java)
* [JavaScript](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingStream.js)
* [Python2/3](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingStream.py)

2
doc/dart-target.md
View File

@ -1,6 +1,6 @@
# ANTLR4 Runtime for Dart
From version 4.9.3 onwards antlr's dart generated code is null sound safety compatible and sets the minimum dart sdk version to 2.12.0.
From version 4.10 onwards antlr's dart generated code is null sound safety compatible and sets the minimum dart sdk version to 2.12.0.
### First steps

24
doc/getting-started.md
View File

@ -6,7 +6,7 @@ Hi and welcome to the version 4 release of ANTLR! It's named after the fearless
ANTLR is really two things: a tool that translates your grammar to a parser/lexer in Java (or other target language) and the runtime needed by the generated parsers/lexers. Even if you are using the ANTLR Intellij plug-in or ANTLRWorks to run the ANTLR tool, the generated code will still need the runtime library.
The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers. In what follows, I talk about antlr-4.9-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers. In what follows, I talk about antlr-4.10-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see [Integrating ANTLR into Development Systems](https://github.com/antlr/antlr4/blob/master/doc/IDEs.md).
@ -16,22 +16,22 @@ If you are going to integrate ANTLR into your existing build system using mvn, a
1. Download
```
$ cd /usr/local/lib
$ curl -O https://www.antlr.org/download/antlr-4.9-complete.jar
$ curl -O https://www.antlr.org/download/antlr-4.10-complete.jar
```
Or just download in browser from website:
[https://www.antlr.org/download.html](https://www.antlr.org/download.html)
and put it somewhere rational like `/usr/local/lib`.
2. Add `antlr-4.9-complete.jar` to your `CLASSPATH`:
2. Add `antlr-4.10-complete.jar` to your `CLASSPATH`:
```
$ export CLASSPATH=".:/usr/local/lib/antlr-4.9-complete.jar:$CLASSPATH"
$ export CLASSPATH=".:/usr/local/lib/antlr-4.10-complete.jar:$CLASSPATH"
```
It's also a good idea to put this in your `.bash_profile` or whatever your startup script is.
3. Create aliases for the ANTLR Tool, and `TestRig`.
```
$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.9-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.9-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.10-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.10-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
```
### WINDOWS
@ -39,13 +39,13 @@ $ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.9-complete.jar:$CLASSPAT
(*Thanks to Graham Wideman*)
0. Install Java (version 1.7 or higher)
1. Download antlr-4.9-complete.jar (or whatever version) from [https://www.antlr.org/download/](https://www.antlr.org/download/)
1. Download antlr-4.10-complete.jar (or whatever version) from [https://www.antlr.org/download/](https://www.antlr.org/download/)
Save to your directory for 3rd party Java libraries, say `C:\Javalib`
2. Add `antlr-4.9-complete.jar` to CLASSPATH, either:
2. Add `antlr-4.10-complete.jar` to CLASSPATH, either:
* Permanently: Using System Properties dialog > Environment variables > Create or append to `CLASSPATH` variable
* Temporarily, at command line:
```
SET CLASSPATH=.;C:\Javalib\antlr-4.9-complete.jar;%CLASSPATH%
SET CLASSPATH=.;C:\Javalib\antlr-4.10-complete.jar;%CLASSPATH%
```
3. Create short convenient commands for the ANTLR Tool, and TestRig, using batch files or doskey commands:
* Batch files (in directory in system PATH) antlr4.bat and grun.bat
@ -71,7 +71,7 @@ Either launch org.antlr.v4.Tool directly:
```
$ java org.antlr.v4.Tool
ANTLR Parser Generator Version 4.9
ANTLR Parser Generator Version 4.10
-o ___ specify output directory where all output is generated
-lib ___ specify location of .tokens files
...
@ -80,8 +80,8 @@ ANTLR Parser Generator Version 4.9
or use -jar option on java:
```
$ java -jar /usr/local/lib/antlr-4.9-complete.jar
ANTLR Parser Generator Version 4.9
$ java -jar /usr/local/lib/antlr-4.10-complete.jar
ANTLR Parser Generator Version 4.10
-o ___ specify output directory where all output is generated
-lib ___ specify location of .tokens files
...

4
doc/go-target.md
View File

@ -20,11 +20,11 @@ go get github.com/antlr/antlr4/runtime/Go/antlr
`go get` has no native way to specify a branch or commit. So, when you run it, you'll download the latest commits. This may or may not be your preference.
You'll need to use git to set the release. For example, to set the release tag for release 4.6.0:
You'll need to use git to set the release. For example, to set the release tag for release 4.9.3:
```bash
cd $GOPATH/src/github.com/antlr/antlr4 # enter the antlr4 source directory
git checkout tags/4.6.0 # the go runtime was added in release 4.6.0
git checkout tags/4.9.3 # the go runtime was added in release 4.9.3
```
A complete list of releases can be found on [the release page](https://github.com/antlr/antlr4/releases).

2
doc/index.md
View File

@ -57,8 +57,6 @@ For those using Java, here's a great [set of ANTLR in Intellij notes](https://do
* [Parsing binary streams](parsing-binary-files.md)
* [Case-Insensitive Lexing](case-insensitive-lexing.md)
* [Parser and lexer interpreters](interpreters.md)
* [Resources](resources.md)

8
doc/java-target.md
View File

@ -134,7 +134,7 @@ Edit the pom.xml file. Now we need to extensively modify the pom.xml file. The f
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
<version>4.5</version>
<version>4.9.3</version>
</dependency>
<dependency>
<groupId>junit</groupId>
@ -150,8 +150,8 @@ Edit the pom.xml file. Now we need to extensively modify the pom.xml file. The f
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<!-- Plugin to compile the g4 files ahead of the java files
@ -164,7 +164,7 @@ Edit the pom.xml file. Now we need to extensively modify the pom.xml file. The f
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>4.5</version>
<version>4.9.3</version>
<executions>
<execution>
<goals>

13
doc/lexer-rules.md
View File

@ -308,3 +308,16 @@ As of 4.5, you can also define channel names like enumerations with the followin
```
channels { WSCHANNEL, MYHIDDEN }
```
## Lexer Rule Options
### caseInsensitive
Defines if the current lexer rule is case-insensitive.
The argument can be `true` or `false`.
The option rewrites `caseInsensitive` grammar option value if it's defined.
```g4
options { caseInsensitive=true; }
STRING options { caseInsensitive=false; } : 'N'? '\'' (~'\'' | '\'\'')* '\''; // lower n is not allowed
```

128
doc/listeners.md
View File

@ -36,3 +36,131 @@ ParseTreeWalker.DEFAULT.walk(extractor, tree); // initiate walk of tree with lis
Listeners and visitors are great because they keep application-specific code out of grammars, making grammars easier to read and preventing them from getting entangled with a particular application.
See the book for more information on listeners and to learn how to use visitors. (The biggest difference between the listener and visitor mechanisms is that listener methods are called independently by an ANTLR-provided walker object, whereas visitor methods must walk their children with explicit visit calls. Forgetting to invoke visitor methods on a node’s children, means those subtrees don’t get visited.)
## Listening during the parse
We can also use listeners to execute code during the parse instead of waiting for a tree walker walks the resulting parse tree. Let's say we have the following simple expression grammar.
```
grammar CalcNoLR;
s : expr EOF ;
expr: add ((MUL | DIV) add)* ;
add : atom ((ADD | SUB) atom)* ;
atom : INT ;
INT : [0-9]+;
MUL : '*';
DIV : '/';
ADD : '+';
SUB : '-';
WS : [ \t]+ -> channel(HIDDEN);
```
We can create a listener that executes during the parse by implementing the listener interface as before:
```java
class CountListener extends CalcNoLRBaseListener {
public int nums = 0;
public boolean execExitS = false;
@Override
public void exitS(CalcNoLRParser.SContext ctx) {
execExitS = true;
}
@Override
public void exitAtom(CalcNoLRParser.AtomContext ctx) {
nums++;
}
}
```
And then passing it to `addParseListener()`:
```java
String input = "2 + 8 / 2";
CalcNoLRLexer lexer = new CalcNoLRLexer(new ANTLRInputStream(input));
CalcNoLRParser parser = new CalcNoLRParser(new CommonTokenStream(lexer));
CountListener counter = new CountListener();
parser.addParseListener(counter);
// Check that the purses valid first
CalcNoLRParser.SContext context = parser.s();
String parseTreeS = context.toStringTree(parser);
assertEquals("(s (expr (add (atom 2) + (atom 8)) / (add (atom 2))) <EOF>)", parseTreeS);
assertEquals(3, counter.nums);
assertEquals(true, counter.execExitS);
```
One should not do very complicated work during the parse because the parser is throwing exception to handle syntax tears. If you're complicated code throws different kind of exception it will screw up the parsing and things will go nuts. If you want to catch and properly handle exceptions in your listener code during the parse, you should override this method from `Parser`:
```java
protected boolean listenerExceptionOccurred = false;
/**
* Notify any parse listeners of an exit rule event.
*
* @see #addParseListener
*/
@override
protected void triggerExitRuleEvent() {
if ( listenerExceptionOccurred ) return;
try {
// reverse order walk of listeners
for (int i = _parseListeners.size() - 1; i >= 0; i--) {
ParseTreeListener listener = _parseListeners.get(i);
_ctx.exitRule(listener);
listener.exitEveryRule(_ctx);
}
}
catch (Throwable e) {
// If an exception is thrown in the user's listener code, we need to bail out
// completely out of the parser, without executing anymore user code. We
// must also stop the parse otherwise other listener actions will attempt to execute
// almost certainly with invalid results. So, record the fact an exception occurred
listenerExceptionOccurred = true;
throw e;
}
}
```
Now, if you throw an exception inside one of the listener methods:
```java
// Now throw an exception in the listener
class ErrorListener extends CalcNoLRBaseListener {
public boolean execExitS = false;
public boolean execExitAtom = false;
@Override
public void exitS(CalcNoLRParser.SContext ctx) {
execExitS = true;
}
@Override
public void exitAtom(CalcNoLRParser.AtomContext ctx) {
execExitAtom = true;
throw new NullPointerException("bail out");
}
}
```
then the exception will properly cause the parser to bailout and the exception will not be thrown out:
```
java.lang.NullPointerException: bail out
at org.antlr.v4.test.runtime.java.api.TestParseListener$2ErrorListener.exitAtom(TestParseListener.java:102)
at org.antlr.v4.test.runtime.java.api.CalcNoLRParser$AtomContext.exitRule(CalcNoLRParser.java:311)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:412)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:654)
at org.antlr.v4.test.runtime.java.api.CalcNoLRParser.atom(CalcNoLRParser.java:336)
at org.antlr.v4.test.runtime.java.api.CalcNoLRParser.add(CalcNoLRParser.java:261)
at org.antlr.v4.test.runtime.java.api.CalcNoLRParser.expr(CalcNoLRParser.java:181)
at org.antlr.v4.test.runtime.java.api.CalcNoLRParser.s(CalcNoLRParser.java:123)
```

67
doc/options.md
View File

@ -12,7 +12,10 @@ where a value can be an identifier, a qualified identifier (for example, a.b.c),
All grammars can use the following options. In combined grammars, all options except language pertain only to the generated parser. Options may be set either within the grammar file using the options syntax (described above) or when invoking ANTLR on the command line, using the `-D` option. (see Section 15.9, [ANTLR Tool Command Line Options](tool-options.md).) The following examples demonstrate both mechanisms; note that `-D` overrides options within the grammar.
* `superClass`. Set the superclass of the generated parser or lexer. For combined grammars, it sets the superclass of the parser.
### `superClass`
Set the superclass of the generated parser or lexer. For combined grammars, it sets the superclass of the parser.
```
$ cat Hi.g4
grammar Hi;
@ -23,12 +26,20 @@ public class HiParser extends XX {
$ grep 'public class' HiLexer.java
public class HiLexer extends Lexer {
```
* `language` Generate code in the indicated language, if ANTLR is able to do so. Otherwise, you will see an error message like this:
### `language`
Generate code in the indicated language, if ANTLR is able to do so. Otherwise, you will see an error message like this:
```
$ antlr4 -Dlanguage=C MyGrammar.g4
error(31): ANTLR cannot generate C code as of version 4.0
```
* `tokenVocab` ANTLR assigns token type numbers to the tokens as it encounters them in a file. To use different token type values, such as with a separate lexer, use this option to have ANTLR pull in the <fileextension>tokens</fileextension> file. ANTLR generates a <fileextension>tokens</fileextension> file from each grammar.
### `tokenVocab`
ANTLR assigns token type numbers to the tokens as it encounters them in a file. To use different token type values, such as with a separate lexer, use this option to have ANTLR pull in the <fileextension>tokens</fileextension> file. ANTLR generates a <fileextension>tokens</fileextension> file from each grammar.
```
$ cat SomeLexer.g4
lexer grammar SomeLexer;
@ -48,7 +59,11 @@ B=3
C=4
ID=1
```
* `TokenLabelType` ANTLR normally uses type <class>Token</class> when it generates variables referencing tokens. If you have passed a <class>TokenFactory</class> to your parser and lexer so that they create custom tokens, you should set this option to your specific type. This ensures that the context objects know your type for fields and method return values.
### `TokenLabelType`
ANTLR normally uses type <class>Token</class> when it generates variables referencing tokens. If you have passed a <class>TokenFactory</class> to your parser and lexer so that they create custom tokens, you should set this option to your specific type. This ensures that the context objects know your type for fields and method return values.
```
$ cat T2.g4
grammar T2;
@ -58,19 +73,45 @@ $ antlr4 T2.g4
$ grep MyToken T2Parser.java
public MyToken x;
```
* `contextSuperClass`. Specify the super class of parse tree internal nodes. Default is `ParserRuleContext`. Should derive from ultimately `RuleContext` at minimum.
Java target can use `contextSuperClass=org.antlr.v4.runtime.RuleContextWithAltNum` for convenience. It adds a backing field for `altNumber`, the alt matched for the associated rule node.
### `contextSuperClass`
Specify the super class of parse tree internal nodes. Default is `ParserRuleContext`. Should derive from ultimately `RuleContext` at minimum.
Java target can use `contextSuperClass=org.antlr.v4.runtime.RuleContextWithAltNum` for convenience. It adds a backing field for `altNumber`, the alt matched for the associated rule node.
### `caseInsensitive`
As of 4.10, ANTLR supports case-insensitive lexers using a grammar option. For example, the parser from the following grammar:
```g4
lexer grammar L;
options { caseInsensitive = true; }
ENGLISH_TOKEN: [a-z]+;
GERMAN_TOKEN: [äéöüß]+;
FRENCH_TOKEN: [àâæ-ëîïôœùûüÿ]+;
CROATIAN_TOKEN: [ćčđšž]+;
ITALIAN_TOKEN: [àèéìòù]+;
SPANISH_TOKEN: [áéíñóúü¡¿]+;
GREEK_TOKEN: [α-ω]+;
RUSSIAN_TOKEN: [а-я]+;
WS: [ ]+ -> skip;
```
matches words such as the following:
```
abcXYZ äéöüßÄÉÖÜß àâæçÙÛÜŸ ćčđĐŠŽ àèéÌÒÙ áéÚÜ¡¿ αβγΧΨΩ абвЭЮЯ
```
ANTLR considers only one-length chars in all cases. For instance, german lower `ß` is not treated as upper `ss` and vice versa.
The mechanism works by automatically transforming grammar references to characters to there upper/lower case equivalent; e.g., `a` to `[aA]`. This means that you do not need to convert your input characters to uppercase--token text will be as it appears in the input stream.
## Rule Options
There are currently no valid rule-level options, but the tool still supports the following syntax for future use:
### caseInsensitive
```
rulename
options {...}
: ...
;
```
The tool support `caseInsensitive` lexer rule option that is described in [lexer-rules.md](lexer-rules.md#caseinsensitive).
## Rule Element Options

16
doc/predicates.md
View File

@ -136,7 +136,7 @@ If, on the other hand, the next character after input `enum` is a letter, then o
Predicates come into play by pruning the set of viable lexer rules. When the lexer encounters a false predicate, it deactivates that rule just like parsers deactivate alternatives with false predicates.
Like parser predicates, lexer predicates can't depend on side effects from lexer actions. That's because actions can only execute after the lexer positively identifies the rule to match. Since predicates are part of the rule selection process, they can't rely on action side effects. Lexer actions must appear after predicates in lexer rules. As an example, here's another way to match enum as a keyword in the lexer:
Like parser predicates, lexer predicates can't depend on side effects from lexer actions. That said, the predicate can depend on a side effect of an action that occured during the recognition of the previous token. That's because actions can only execute after the lexer positively identifies the rule to match. Since predicates are part of the rule selection process, they can't rely on action side effects created by actions in currently-prospective rules. Lexer actions must appear after predicates in lexer rules. As an example, here's another way to match enum as a keyword in the lexer:
```
ENUM: [a-z]+ {getText().equals("enum")}?
@ -162,3 +162,17 @@ That works great, but it's really just for instructional purposes. It's easier t
```
ENUM : 'enum' ;
```
Here's another example of a predicate. It's important to note that the predicate is evaluated before the action because actions are only executed if the lexer rule matches. The actions are not executed in line; they are collected and executed en mass later.
```
INDENT : [ \t]+ {System.out.println("INDENT")>} {this.getCharPositionInLine()==0}? ;
```
For more information on how actions and predicates operate in the lexer, see [Lexer actions and semantic predicates are executed out of order](https://github.com/antlr/antlr4/issues/3611) and [Lexer.getCharIndex() return value not behaving as expected](https://github.com/antlr/antlr4/issues/3606). The lexer rule that will not work as expected is:
```
Stuff : ( 'a'+ {count++;} | 'b') 'c' 'd' {count == 3}? ;
```
The `count++` code we'll not execute until after `Stuff` has been recognized (assuming count!=3).

376
doc/releasing-antlr.md
View File

@ -2,108 +2,97 @@
## Github
Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1).
Create a pre-release or full release at github; e.g., [tag 4.9](https://github.com/antlr/antlr4/tree/4.9).
### Turn on DCO Enforcement
As of 4.10, we will be using the Linux DCO not the previous contributors license agreement that required signing the file. Now, we use the DCO and contributors must use `-s` on each commit to the branch associated with a pull request.
See [GitHub App DCO](https://github.com/apps/dco).
Make sure this feature is turned on for the `dev` and `master` branch upon release.
### Delete existing release tag
Wack any existing tag as mvn will create one and it fails if already there.
```
$ git tag -d 4.9
$ git push origin :refs/tags/4.9
$ git push upstream :refs/tags/4.9
$ git tag -d 4.10
$ git push origin :refs/tags/4.10
$ git push upstream :refs/tags/4.10
```
### Create release candidate tag
```bash
$ git tag -a 4.9-rc1 -m 'heading towards 4.9'
$ git push origin 4.9-rc1
$ git push upstream 4.9-rc1
$ git tag -a 4.10-rc.1 -m 'heading towards 4.10'
$ git push origin 4.10-rc.1
$ git push upstream 4.10-rc1
```
## Copy PHP runtime over
### Go release tags
Bump version to 4.9 in `~/antlr/code/antlr-php-runtime/src/RuntimeMetaData.php` in separate repository and commit plus push.
It seems that [Go needs a `v` in the release git tag](https://go.dev/ref/mod#glos-version) so make sure that we double up with 4.10 and v4.10.
```
cd ~/antlr/code/antlr-php-runtime/src
... vi RuntimeMetaData.php ...
git commit -a -m "Update PHP Runtime to latest version"
## Bump version in code and other files
There are a number of files that require inversion number be updated.
(In a `pred-4.10` branch) Here is a simple script to display any line from the critical files with, say, `4.10` in it. Here's an example run of the script:
```bash
~/antlr/code/antlr4 $ python scripts/update_antlr_version.py 4.9.3 4.10
Updating ANTLR version from 4.9.3 to 4.10
Set ANTLR repo root (default ~/antlr/code/antlr4):
Perform antlr4 `mvn clean` and wipe build dirs Y/N? (default no):
Ok, not cleaning antlr4 dir
4.9.3 appears on 2 lines so _not_ updating /tmp/antlr4/runtime/JavaScript/package-lock.json
4.9.3 not in /tmp/antlr4/doc/releasing-antlr.md
```
them back over in the ANTLR repo:
```
cd runtime
rm -rf PHP
mkdir PHP
cp ~/antlr/code/antlr-php-runtime/LICENSE PHP
cp ~/antlr/code/antlr-php-runtime/README.md PHP
cp ~/antlr/code/antlr-php-runtime/composer.json PHP
cp ~/antlr/code/antlr-php-runtime/phpcs.xml.dist PHP
cp ~/antlr/code/antlr-php-runtime/phpstan.neon.dist PHP
cp -r ~/antlr/code/antlr-php-runtime/src PHP
```
## Bump version
Edit the repository looking for 4.5 or whatever and update it. Bump version in the following files:
* runtime/Java/src/org/antlr/v4/runtime/RuntimeMetaData.java
* runtime/Python2/setup.py
* runtime/Python2/src/antlr4/Recognizer.py
* runtime/Python3/setup.py
* runtime/Python3/src/antlr4/Recognizer.py
* runtime/CSharp/src/Antlr4.csproj
* runtime/PHP/src/RuntimeMetaData.php
* runtime/JavaScript/package.json
* runtime/JavaScript/src/antlr4/Recognizer.js
* runtime/JavaScript/package-lock.json
* runtime/Cpp/VERSION
* runtime/Cpp/runtime/src/RuntimeMetaData.cpp
* runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake
* runtime/Cpp/demo/generate.cmd
* runtime/Go/antlr/recognizer.go
* runtime/Swift/Antlr4/org/antlr/v4/runtime/RuntimeMetaData.swift
* runtime/Dart/lib/src/runtime_meta_data.dart
* runtime/Dart/pubspec.yaml
* runtime/Swift/Tests/Antlr4Tests/RuntimeMetaDataTests.swift
* runtime/Swift/Sources/Antlr4/RuntimeMetaData.swift
* runtime/CSharp/src/Tree/Xpath/XPathLexer.cs
* runtime/CSharp/src/README.md
* runtime/CSharp/src/Properties/AssemblyInfo.cs
* tool/src/org/antlr/v4/codegen/target/GoTarget.java
* tool/src/org/antlr/v4/codegen/target/CppTarget.java
* tool/src/org/antlr/v4/codegen/target/CSharpTarget.java
* tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java
* tool/src/org/antlr/v4/codegen/target/Python2Target.java
* tool/src/org/antlr/v4/codegen/target/Python3Target.java
* tool/src/org/antlr/v4/codegen/target/SwiftTarget.java
* tool/src/org/antlr/v4/codegen/target/PHPTarget.java
* tool/src/org/antlr/v4/codegen/Target.java
* tool/resources/org/antlr/v4/tool/templates/codegen/Swift/Swift.stg
Here is a simple script to display any line from the critical files with, say, `4.9` in it:
It's also worth doing a quick check to see if you find any other references to a version:
```bash
mvn clean
rm -rf runtime/CSharp/src/bin
rm -rf runtime/CSharp/src/obj
rm -rf runtime/Cpp/runtime/build
rm -rf runtime/gen
rm -rf runtime/JavaScript/dist
find tool runtime -type f -exec grep -l '4\.9' {} \;
find . -type f -exec grep -l '4\.9' {} \; | grep -v -E '\.o|\.a|\.jar|\.dylib|node_modules/|\.class|tests/|CHANGELOG|\.zip|\.gz|.iml|.svg'
```
Commit to repository.
## Building
### PHP runtime
ugh. apparently you have to `mvn install` and then `mvn compile` or some such or subdir pom.xml's won't see the latest runtime build.
We only have to copy the PHP runtime into the ANTLR repository to run the unittests. But, we still need to bump the version to 4.10 in `~/antlr/code/antlr-php-runtime/src/RuntimeMetaData.php` in the separate repository, commit, and push.
```
cd ~/antlr/code/antlr-php-runtime/src
git checkout dev # Should be the default
... vi RuntimeMetaData.php ...
git commit -a -m "Update PHP Runtime to latest version"
```
## Build XPath parsers
This section addresses a [circular dependency regarding XPath](https://github.com/antlr/antlr4/issues/3600). In the java target I avoided a circular dependency (gen 4.10 parser for XPath using 4.10 which needs it to build) by hand building the parser: runtime/Java/src/org/antlr/v4/runtime/tree/xpath/XPath.java. Probably we won't have to rerun this for the patch releases, just major ones that alter the ATN serialization.
```
cd ~/antlr/code/antlr4/runtime/CSharp/src/Tree/Xpath
java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.10-SNAPSHOT/antlr4-4.10-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=CSharp XPathLexer.g4
cd ~/antlr/code/antlr4/runtime/Python3/tests/expr
java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.10-SNAPSHOT/antlr4-4.10-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python2 Expr.g4
java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.10-SNAPSHOT/antlr4-4.10-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python2 XPathLexer.g4
cd ~/antlr/code/antlr4/runtime/Python3/tests/expr
java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.10-SNAPSHOT/antlr4-4.10-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python3 Expr.g4
java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.10-SNAPSHOT/antlr4-4.10-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python3 XPathLexer.g4
```
## Maven Repository Settings
ugh. apparently you have to `mvn install` and then `mvn compile` or some such or subdir pom.xml's won't see the latest runtime build.
First, make sure you have maven set up to communicate with staging servers etc... Create file `~/.m2/settings.xml` with appropriate username/password for staging server and gpg.keyname/passphrase for signing. Make sure it has strict visibility privileges to just you. On unix, it looks like:
```bash
@ -149,7 +138,7 @@ Here is the file template
## Maven deploy snapshot
The goal is to get a snapshot, such as `4.9-SNAPSHOT`, to the staging server: [antlr4 tool](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4) and [antlr4 java runtime](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-runtime).
The goal is to get a snapshot, such as `4.10-SNAPSHOT`, to the staging server: [antlr4 tool](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4/4.10-SNAPSHOT/) and [antlr4 java runtime](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-runtime/4.10-SNAPSHOT/).
Do this:
@ -158,15 +147,15 @@ $ mvn install -DskipTests # seems required to get the jar files visible to mave
$ mvn deploy -DskipTests
...
[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ antlr4-tool-testsuite ---
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/maven-metadata.xml
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/antlr4-tool-testsuite-4.9-20161211.173752-1.jar
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/antlr4-tool-testsuite-4.9-20161211.173752-1.jar (3 KB at 3.4 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/antlr4-tool-testsuite-4.9-20161211.173752-1.pom
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/antlr4-tool-testsuite-4.9-20161211.173752-1.pom (3 KB at 6.5 KB/sec)
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/maven-metadata.xml
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/antlr4-tool-testsuite-4.10-20161211.173752-1.jar
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/antlr4-tool-testsuite-4.10-20161211.173752-1.jar (3 KB at 3.4 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/antlr4-tool-testsuite-4.10-20161211.173752-1.pom
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/antlr4-tool-testsuite-4.10-20161211.173752-1.pom (3 KB at 6.5 KB/sec)
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml
Downloaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml (371 B at 1.4 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/maven-metadata.xml
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.9-SNAPSHOT/maven-metadata.xml (774 B at 1.8 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/maven-metadata.xml
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.10-SNAPSHOT/maven-metadata.xml (774 B at 1.8 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml (388 B at 0.9 KB/sec)
[INFO] ------------------------------------------------------------------------
@ -178,7 +167,7 @@ Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antl
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 6.547 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 2.519 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 2.385 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [ 15.276 s]
[INFO] ANTLR 4 Runtime Tests (3rd generation) ............. SUCCESS [ 15.276 s]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 2.233 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
@ -194,60 +183,47 @@ Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antl
The maven deploy lifecycle phased deploys the artifacts and the poms for the ANTLR project to the [sonatype remote staging server](https://oss.sonatype.org/content/repositories/snapshots/).
```bash
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn deploy -DskipTests
mvn deploy -DskipTests
```
With JDK 1.7 (not 6 or 8), do this:
Make sure `gpg` is installed (`brew install gpg` on mac). Also must [create a key and publish it](https://blog.sonatype.com/2010/01/how-to-generate-pgp-signatures-with-maven/) then update `.m2/settings` to use that public key.
Then:
```bash
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn release:prepare -Darguments="-DskipTests"
mvn release:prepare -Darguments="-DskipTests"
```
Hm...per https://github.com/keybase/keybase-issues/issues/1712 we need this to make gpg work:
Hmm...per https://github.com/keybase/keybase-issues/issues/1712 we need this to make gpg work:
```bash
export GPG_TTY=$(tty)
```
Side note to set jdk 1.7 on os x:
You should see 0x37 in generated .class files after 0xCAFEBABE; see [Java SE 11 = 55 (0x37 hex)](https://en.wikipedia.org/wiki/Java_class_file):
```bash
alias java="`/usr/libexec/java_home -v 1.7`/bin/java"
alias javac="`/usr/libexec/java_home -v 1.7`/bin/javac"
alias javadoc="`/usr/libexec/java_home -v 1.7`/bin/javadoc"
alias jar="`/usr/libexec/java_home -v 1.7`/bin/jar"
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`
```
But I think just this on front of mvn works:
```
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn ...
```
You should see 0x33 in generated .class files after 0xCAFEBABE; see [Java SE 7 = 51 (0x33 hex)](https://en.wikipedia.org/wiki/Java_class_file):
```bash
beast:/tmp/org/antlr/v4 $ od -h Tool.class |head -1
0000000 feca beba 0000 3300 fa04 0207 0ab8 0100
~/antlr/code/antlr4 $ od -h tool/target/classes/org/antlr/v4/Tool.class |head -1
0000000 feca beba 0000 3700 ed04 0207 0a9d 0100
^^
```
It will start out by asking you the version number:
```
...
What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.9: : 4.9
What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.9: :
What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.9: :
What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.9: :
What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.9: :
What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.9: :
What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.9: : 4.9
What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.9.1-SNAPSHOT:
What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.10: : 4.10
What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.10: :
What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.10: :
What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.10: :
What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.10: :
What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.10: :
What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.10: : 4.10
What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.10.1-SNAPSHOT:
...
```
Maven will go through your pom.xml files to update versions from 4.9-SNAPSHOT to 4.9 for release and then to 4.9.1-SNAPSHOT after release, which is done with:
Maven will go through your pom.xml files to update versions from 4.10-SNAPSHOT to 4.10 for release and then to 4.10.1-SNAPSHOT after release, which is done with:
```bash
mvn release:perform -Darguments="-DskipTests"
@ -261,42 +237,14 @@ Now, go here:
and on the left click "Staging Repositories". You click the staging repo and close it, then you refresh, click it and release it. It's done when you see it here:
&nbsp;&nbsp;&nbsp;&nbsp;[https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.9-1/antlr4-runtime-4.9-1.jar](https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.9-1/antlr4-runtime-4.9-1.jar)
&nbsp;&nbsp;&nbsp;&nbsp;[https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.10/antlr4-runtime-4.10.jar](https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.10/antlr4-runtime-4.10.jar)
All releases should be here: https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/
Copy the jars to antlr.org site and update download/index.html
```bash
cp ~/.m2/repository/org/antlr/antlr4-runtime/4.9/antlr4-runtime-4.9.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.9.jar
cp ~/.m2/repository/org/antlr/antlr4/4.9/antlr4-4.9-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.9-complete.jar
cd ~/antlr/sites/website-antlr4/download
git add antlr-4.9-complete.jar
git add antlr-runtime-4.9.jar
```
Update on site:
* download.html
* index.html
* api/index.html
* download/index.html
* scripts/topnav.js
```
git commit -a -m 'add 4.9 jars'
git push origin gh-pages
```
All releases should be here: [https://repo1.maven.org/maven2/org/antlr/antlr4-runtime](https://repo1.maven.org/maven2/org/antlr/antlr4-runtime).
## Deploying Targets
### JavaScript
```bash
cd runtime/JavaScript
# git add, commit, push
```
**Push to npm**
```bash
@ -304,7 +252,7 @@ cd runtime/JavaScript
npm update
npm install
npm run build
npm login
npm login # asks for username/password/2FA (npmjs.com)
npm publish # don't put antlr4 on there or it will try to push the old version for some reason
```
@ -312,55 +260,65 @@ Move (and zip) target to website:
```bash
cd src
zip -r ~/antlr/sites/website-antlr4/download/antlr-javascript-runtime-4.9.zip .
zip -r ~/antlr/sites/website-antlr4/download/antlr-javascript-runtime-4.10.zip .
```
### CSharp
Now we have [appveyor create artifact](https://ci.appveyor.com/project/parrt/antlr4/build/artifacts). Go to [nuget](https://www.nuget.org/packages/manage/upload) to upload the `.nupkg`.
### Publishing to Nuget from Windows
**Install the pre-requisites**
Of course you need Mono and `nuget` to be installed. On mac:
You need Mono and `nuget` to be installed. On mac:
- .NET build tools - can be loaded from [here](https://www.visualstudio.com/downloads/) (I need dotnet 5 and 3.1 versions)
- nuget - download [nuget.exe](https://www.nuget.org/downloads)
- dotnet - follow [the instructions here](https://www.microsoft.com/net/core)
From @kvanTTT: Install `dotnet` on any platform (see https://dotnet.microsoft.com/download) and run the following command on any OS (Win, Linux, macOS):
* building: `dotnet build runtime/CSharp/src/Antlr4.csproj -c Release`
Output `.dll` will be in `runtime/CSharp/src/bin/Release/netstandard2.0` or in `runtime/CSharp/src/bin/Release/netstandard2.1`
* packing: `dotnet pack runtime/CSharp/src/Antlr4.csproj -c Release`
Output `.nupkg` will be in `runtime/CSharp/src/bin/Release/Antlr4.Runtime.Standard.4.9.1.nupkg`
Alternatively, you can install Visual Studio 2017 and make sure to check boxes with .NET Core SDK.
You also need to enable .NET Framework 3.5 support in Windows "Programs and Features".
If everything is ok, the following command will restore nuget packages, build Antlr for .NET Standard and .NET 3.5 and create nuget package:
**Creating the signed assembly**
```PS
msbuild /target:restore /target:rebuild /target:pack /property:Configuration=Release .\Antlr4.dotnet.sln /verbosity:minimal
```
From Mac:
This should display something like this:
**Creating and packaging the assembly**
```
Microsoft (R) Build Engine version 15.4.8.50001 for .NET Framework
```bash
critter:~ $ cd ~/antlr/code/antlr4/runtime/CSharp/src
critter:~/antlr/code/antlr4/runtime/CSharp/src $ dotnet build -c Release Antlr4.csproj
Microsoft (R) Build Engine version 16.7.2+b60ddb6f4 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
Restoring packages for C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Antlr4.Runtime.dotnet.csproj...
Generating MSBuild file C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\obj\Antlr4.Runtime.dotnet.csproj.nuget.g.props.
Generating MSBuild file C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\obj\Antlr4.Runtime.dotnet.csproj.nuget.g.targets.
Restore completed in 427.62 ms for C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Antlr4.Runtime.dotnet.csproj.
Antlr4.Runtime.dotnet -> C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\netstandard1.3\Antlr4.Runtime.Standard.dll
Antlr4.Runtime.dotnet -> C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\net35\Antlr4.Runtime.Standard.dll
Successfully created package 'C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\Antlr4.Runtime.Standard.4.9.3.nupkg'.
Determining projects to restore...
Restored /Users/parrt/antlr/code/antlr4/runtime/CSharp/src/Antlr4.csproj (in 340 ms).
Antlr4 -> /Users/parrt/antlr/code/antlr4/runtime/CSharp/src/bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll
Successfully created package '/Users/parrt/antlr/code/antlr4/runtime/CSharp/src/bin/Release/Antlr4.Runtime.Standard.4.10.0.nupkg'.
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:06.77
critter:~/antlr/code/antlr4/runtime/CSharp/src $ /Library/Frameworks/Mono.framework/Versions/Current/Commands/sn -R bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll Antlr4.snk
Mono StrongName - version 6.12.0.0
StrongName utility for signing assemblies
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Assembly bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll signed.
critter:~/antlr/code/antlr4/runtime/CSharp/src $ /Library/Frameworks/Mono.framework/Versions/Current/Commands/sn -v bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll
Mono StrongName - version 6.12.0.0
StrongName utility for signing assemblies
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Assembly bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll is strongnamed.
$ tree /Users/parrt/antlr/code/antlr4/runtime/CSharp/src/bin/Release/
/Users/parrt/antlr/code/antlr4/runtime/CSharp/src/bin/Release/
├── Antlr4.Runtime.Standard.4.10.0.nupkg
└── netstandard2.0
├── Antlr4.Runtime.Standard.deps.json
├── Antlr4.Runtime.Standard.dll
├── Antlr4.Runtime.Standard.pdb
└── Antlr4.Runtime.Standard.xml
1 directory, 5 files
```
**Publishing to NuGet**
@ -376,6 +334,7 @@ nuget push Antlr4.Runtime.Standard.<version>.nupkg <your-key> -Source https://ww
Nuget packages are also accessible as artifacts of [AppVeyor builds](https://ci.appveyor.com/project/parrt/antlr4/build/artifacts).
### Python
The Python targets get deployed with `setup.py`. First, set up `~/.pypirc` with tight privileges:
@ -436,7 +395,7 @@ On a Mac (with XCode 7+ installed):
```bash
cd runtime/Cpp
./deploy-macos.sh
cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.9-macos.zip
cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.10-macos.zip
```
On any Mac or Linux machine:
@ -444,7 +403,7 @@ On any Mac or Linux machine:
```bash
cd runtime/Cpp
./deploy-source.sh
cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.9-source.zip
cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.10-source.zip
```
On a Windows machine the build scripts checks if VS 2017 and/or VS 2019 are installed and builds binaries for each, if found. This script requires 7z to be installed (http://7-zip.org then do `set PATH=%PATH%;C:\Program Files\7-Zip\` from DOS not powershell).
@ -452,17 +411,16 @@ On a Windows machine the build scripts checks if VS 2017 and/or VS 2019 are inst
```bash
cd runtime/Cpp
deploy-windows.cmd Community
cp antlr4-cpp-runtime-vs2019.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.9-vs2019.zip
cp antlr4-cpp-runtime-vs2019.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.10-vs2019.zip
```
Move target to website (**_rename to a specific ANTLR version first if needed_**):
```bash
pushd ~/antlr/sites/website-antlr4/download
# vi index.html
git add antlr4-cpp-runtime-4.9-macos.zip
git add antlr4-cpp-runtime-4.9-windows.zip
git add antlr4-cpp-runtime-4.9-source.zip
git add antlr4-cpp-runtime-4.10-macos.zip
git add antlr4-cpp-runtime-4.10-windows.zip
git add antlr4-cpp-runtime-4.10-source.zip
git commit -a -m 'update C++ runtime'
git push origin gh-pages
popd
@ -483,21 +441,48 @@ It will warn that no change log found for the new version.
If there are changes relevant to dart in this release, edit [CHANGELOG.md](https://github.com/antlr/antlr4/blob/master/runtime/Dart/CHANGELOG.md) to describe the changes.
Otherwise enter `N` to ignore the warning.
## Update javadoc for runtime and tool
## Update website
Above build should make latest in
```
~/.m2/repository/org/antlr/antlr4-runtime/4.9/antlr4-runtime-4.9
```
but you can regen (watch pom version!):
### javadoc for runtime and tool
```bash
$ cd antlr4
$ mvn -DskipTests javadoc:jar install
cd ~/antlr/code/antlr4
mvn -DskipTests javadoc:jar -q install # get lots of errors but works
```
Jars are in:
```
~/.m2/repository/org/antlr/antlr4-runtime/4.10/antlr4-runtime-4.10
```
### Update version and copy jars / api
Copy javadoc and java jars to website using this script:
```bash
cd ~/antlr/code/antlr4
python scripts/deploy.py 4.9.3 4.10
```
<!--
```bash
cp ~/.m2/repository/org/antlr/antlr4-runtime/4.10/antlr4-runtime-4.10.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.10.jar
cp ~/.m2/repository/org/antlr/antlr4/4.10/antlr4-4.10-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.10-complete.jar
cd ~/antlr/sites/website-antlr4/download
git add antlr-4.10-complete.jar
git add antlr-runtime-4.10.jar
```
-->
Once it's done, you must do the following manually:
```
git commit -a -m 'add 4.10 jars'
git push origin gh-pages
```
<!--
Then copy to website:
```bash
@ -505,12 +490,13 @@ cd ~/antlr/sites/website-antlr4/api
git checkout gh-pages
git pull origin gh-pages
cd Java
jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.9/antlr4-runtime-4.9-javadoc.jar
jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.10/antlr4-runtime-4.10-javadoc.jar
cd ../JavaTool
jar xvf ~/.m2/repository/org/antlr/antlr4/4.9/antlr4-4.9-javadoc.jar
jar xvf ~/.m2/repository/org/antlr/antlr4/4.10/antlr4-4.10-javadoc.jar
git commit -a -m 'freshen api doc'
git push origin gh-pages
```
-->
## Update Intellij plug-in

105
doc/resources/CaseChangingCharStream.cs
View File

@ -1,105 +0,0 @@
/* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
* Use of this file is governed by the BSD 3-clause license that
* can be found in the LICENSE.txt file in the project root.
*/
using System;
using Antlr4.Runtime.Misc;
namespace Antlr4.Runtime
{
/// <summary>
/// This class supports case-insensitive lexing by wrapping an existing
/// <see cref="ICharStream"/> and forcing the lexer to see either upper or
/// lowercase characters. Grammar literals should then be either upper or
/// lower case such as 'BEGIN' or 'begin'. The text of the character
/// stream is unaffected. Example: input 'BeGiN' would match lexer rule
/// 'BEGIN' if constructor parameter upper=true but getText() would return
/// 'BeGiN'.
/// </summary>
public class CaseChangingCharStream : ICharStream
{
private ICharStream stream;
private bool upper;
/// <summary>
/// Constructs a new CaseChangingCharStream wrapping the given <paramref name="stream"/> forcing
/// all characters to upper case or lower case.
/// </summary>
/// <param name="stream">The stream to wrap.</param>
/// <param name="upper">If true force each symbol to upper case, otherwise force to lower.</param>
public CaseChangingCharStream(ICharStream stream, bool upper)
{
this.stream = stream;
this.upper = upper;
}
public int Index
{
get
{
return stream.Index;
}
}
public int Size
{
get
{
return stream.Size;
}
}
public string SourceName
{
get
{
return stream.SourceName;
}
}
public void Consume()
{
stream.Consume();
}
[return: NotNull]
public string GetText(Interval interval)
{
return stream.GetText(interval);
}
public int LA(int i)
{
int c = stream.LA(i);
if (c <= 0)
{
return c;
}
char o = (char)c;
if (upper)
{
return (int)char.ToUpperInvariant(o);
}
return (int)char.ToLowerInvariant(o);
}
public int Mark()
{
return stream.Mark();
}
public void Release(int marker)
{
stream.Release(marker);
}
public void Seek(int index)
{
stream.Seek(index);
}
}
}

81
doc/resources/CaseChangingCharStream.java
View File

@ -1,81 +0,0 @@
package org.antlr.v4.runtime;
import org.antlr.v4.runtime.misc.Interval;
/**
* This class supports case-insensitive lexing by wrapping an existing
* {@link CharStream} and forcing the lexer to see either upper or
* lowercase characters. Grammar literals should then be either upper or
* lower case such as 'BEGIN' or 'begin'. The text of the character
* stream is unaffected. Example: input 'BeGiN' would match lexer rule
* 'BEGIN' if constructor parameter upper=true but getText() would return
* 'BeGiN'.
*/
public class CaseChangingCharStream implements CharStream {
final CharStream stream;
final boolean upper;
/**
* Constructs a new CaseChangingCharStream wrapping the given {@link CharStream} forcing
* all characters to upper case or lower case.
* @param stream The stream to wrap.
* @param upper If true force each symbol to upper case, otherwise force to lower.
*/
public CaseChangingCharStream(CharStream stream, boolean upper) {
this.stream = stream;
this.upper = upper;
}
@Override
public String getText(Interval interval) {
return stream.getText(interval);
}
@Override
public void consume() {
stream.consume();
}
@Override
public int LA(int i) {
int c = stream.LA(i);
if (c <= 0) {
return c;
}
if (upper) {
return Character.toUpperCase(c);
}
return Character.toLowerCase(c);
}
@Override
public int mark() {
return stream.mark();
}
@Override
public void release(int marker) {
stream.release(marker);
}
@Override
public int index() {
return stream.index();
}
@Override
public void seek(int index) {
stream.seek(index);
}
@Override
public int size() {
return stream.size();
}
@Override
public String getSourceName() {
return stream.getSourceName();
}
}

65
doc/resources/CaseChangingStream.js
View File

@ -1,65 +0,0 @@
//
/* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
* Use of this file is governed by the BSD 3-clause license that
* can be found in the LICENSE.txt file in the project root.
*/
//
function CaseChangingStream(stream, upper) {
this._stream = stream;
this._upper = upper;
}
CaseChangingStream.prototype.LA = function(offset) {
var c = this._stream.LA(offset);
if (c <= 0) {
return c;
}
return String.fromCodePoint(c)[this._upper ? "toUpperCase" : "toLowerCase"]().codePointAt(0);
};
CaseChangingStream.prototype.reset = function() {
return this._stream.reset();
};
CaseChangingStream.prototype.consume = function() {
return this._stream.consume();
};
CaseChangingStream.prototype.LT = function(offset) {
return this._stream.LT(offset);
};
CaseChangingStream.prototype.mark = function() {
return this._stream.mark();
};
CaseChangingStream.prototype.release = function(marker) {
return this._stream.release(marker);
};
CaseChangingStream.prototype.seek = function(_index) {
return this._stream.seek(_index);
};
CaseChangingStream.prototype.getText = function(start, stop) {
return this._stream.getText(start, stop);
};
CaseChangingStream.prototype.toString = function() {
return this._stream.toString();
};
Object.defineProperty(CaseChangingStream.prototype, "index", {
get: function() {
return this._stream.index;
}
});
Object.defineProperty(CaseChangingStream.prototype, "size", {
get: function() {
return this._stream.size;
}
});
exports.CaseChangingStream = CaseChangingStream;

13
doc/resources/CaseChangingStream.py
View File

@ -1,13 +0,0 @@
class CaseChangingStream():
def __init__(self, stream, upper):
self._stream = stream
self._upper = upper
def __getattr__(self, name):
return self._stream.__getattribute__(name)
def LA(self, offset):
c = self._stream.LA(offset)
if c <= 0:
return c
return ord(chr(c).upper() if self._upper else chr(c).lower())

64
doc/resources/case_changing_char_stream.dart
View File

@ -1,64 +0,0 @@
// @dart=2.12
import '../../runtime/Dart/lib/antlr4.dart';
import '../../runtime/Dart/lib/src/interval_set.dart';
/// This class supports case-insensitive lexing by wrapping an existing
/// {@link CharStream} and forcing the lexer to see either upper or
/// lowercase characters. Grammar literals should then be either upper or
/// lower case such as 'BEGIN' or 'begin'. The text of the character
/// stream is unaffected. Example: input 'BeGiN' would match lexer rule
/// 'BEGIN' if constructor parameter upper=true but getText() would return
/// 'BeGiN'.
class CaseChangingCharStream extends CharStream {
final CharStream stream;
final bool upper;
/// Constructs a new CaseChangingCharStream wrapping the given [stream] forcing
/// all characters to upper case or lower case depending on [upper].
CaseChangingCharStream(this.stream, this.upper);
@override
int? LA(int i) {
int? c = stream.LA(i);
if (c == null || c <= 0) {
return c;
}
String newCaseStr;
if (upper) {
newCaseStr = String.fromCharCode(c).toUpperCase();
} else {
newCaseStr = String.fromCharCode(c).toLowerCase();
}
// Skip changing case if length changes (e.g., ß -> SS).
if (newCaseStr.length != 1) {
return c;
} else {
return newCaseStr.codeUnitAt(0);
}
}
@override
String get sourceName => stream.sourceName;
@override
void consume() => stream.consume();
@override
String getText(Interval interval) => stream.getText(interval);
@override
int get index => stream.index;
@override
int mark() => stream.mark();
@override
void release(int marker) => stream.release(marker);
@override
void seek(int index) => stream.seek(index);
@override
int get size => stream.size;
}

37
doc/resources/case_changing_stream.go
View File

@ -1,37 +0,0 @@
package antlr_resource
import (
"unicode"
"github.com/antlr/antlr4/runtime/Go/antlr"
)
// CaseChangingStream wraps an existing CharStream, but upper cases, or
// lower cases the input before it is tokenized.
type CaseChangingStream struct {
antlr.CharStream
upper bool
}
// NewCaseChangingStream returns a new CaseChangingStream that forces
// all tokens read from the underlying stream to be either upper case
// or lower case based on the upper argument.
func NewCaseChangingStream(in antlr.CharStream, upper bool) *CaseChangingStream {
return &CaseChangingStream{in, upper}
}
// LA gets the value of the symbol at offset from the current position
// from the underlying CharStream and converts it to either upper case
// or lower case.
func (is *CaseChangingStream) LA(offset int) int {
in := is.CharStream.LA(offset)
if in < 0 {
// Such as antlr.TokenEOF which is -1
return in
}
if is.upper {
return int(unicode.ToUpper(rune(in)))
}
return int(unicode.ToLower(rune(in)))
}

40
doc/swift-target.md
View File

@ -1,15 +1,10 @@
# ANTLR4 Language Target, Runtime for Swift
## Requirements
ANTLR 4.7.2 requires Swift 4.2. It works on Swift 4.2.1 also.
ANTLR 4.7.1 requires Swift 4.0, and does not work on Swift 4.2. (The status of
Swift 4.1 support is unknown.)
## Performance Note
To use ANTLR4 Swift target in production environment, make sure to turn on compiler optimizations by following [these instructions](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#build-configurations) if you use SwiftPM to build your project. If you are using Xcode to build your project, it's unlikely you will not use `release` build for production build.
To use ANTLR4 Swift target in production environment, make sure to turn on compiler optimizations by following [these instructions](https://github.com/apple/swift-package-manager/blob/main/Documentation/Usage.md#setting-the-build-configuration) if you use SwiftPM to build your project.
If you are using Xcode to build your project, it's unlikely you will not use `release` build for production build.
Conclusion is, you need to turn on `release` mode (which will have all the optimization pre configured for you) so the ANTLR4 Swift target can have reasonable parsing speed.
@ -60,10 +55,7 @@ Note that even if you are otherwise using ANTLR from a binary distribution,
you should compile the ANTLR Swift runtime from source, because the Swift
language does not yet have a stable ABI.
ANTLR uses Swift Package Manager to generate Xcode project files. Note that
Swift Package Manager does not currently support iOS, watchOS, or tvOS, so
if you wish to use those platforms, you will need to alter the project build
settings manually as appropriate.
ANTLR uses Swift Package Manager to generate Xcode project files.
#### Download source code for ANTLR
@ -135,30 +127,12 @@ The runtime and generated grammar should now build correctly.
### Swift Package Manager Projects
Since we cannot have a separate repository for Swift target (see issue [#1774](https://github.com/antlr/antlr4/issues/1774)),
and Swift is currently not ABI stable. We currently support SPM-based
projects by creating temporary local repository.
Add Antlr4 as a dependency to your `Package.swift` file. For more information, please see the [Swift Package Manager documentation](https://github.com/apple/swift-package-manager/tree/master/Documentation).
For people using [Swift Package Manager](https://swift.org/package-manager/),
the __boot.py__ script supports generating local repository that can be used
as a dependency to your project. Simply run:
```swift
.package(name: "Antlr4", url: "https://github.com/antlr/antlr4", from: "4.10"
```
python boot.py --gen-spm-module
```
The prompt will show something like below:
<img src=images/gen_spm_module.png width="800">
Put the SPM directive that contains the url to temporary repository to your
project's Package.swift. And run `swift build` in your project.
The project is generated in your system's `/tmp/` directory, if you find it
inconvenient, consider copy that generated ANTLR repository to some place
that won't be cleaned automatically and update `url` parameter in your
`Package.swift` file.
## Swift access levels
You may use the `accessLevel` option to control the access levels on generated

2
docker/Dockerfile
View File

@ -2,7 +2,7 @@ FROM adoptopenjdk/openjdk11:alpine AS builder
WORKDIR /opt/antlr4
ARG ANTLR_VERSION="4.9.3"
ARG ANTLR_VERSION="4.10"
ARG MAVEN_OPTS="-Xmx1G"

19
contributors.txt → historical-contributors-agreement.txt
View File

@ -1,9 +1,5 @@
ANTLR Project Contributors Certification of Origin and Rights
NOTE: This tool is mature and Terence is mostly occupied elsewhere. We
can't accept any changes that could have widespread effects on thousands
of existing projects. Sorry!
All contributors to ANTLR v4 must formally agree to abide by this
certificate of origin by signing on the bottom with their github
userid, full name, email address (you can obscure your e-mail, but it
@ -222,6 +218,7 @@ YYYY/MM/DD, github id, Full name, email
2019/01/01, khoroshilov, Alexey Khoroshilov, khoroshilov@ispras.ru
2019/01/02, wkhemir, Wail Khemir, khemirwail@gmail.com
2019/01/16, kuegi, Markus Zancolo, markus.zancolo@roomle.com
2019/01/29, hc-codersatlas, Harry Chan, harry.chan@codersatlas.com
2019/02/06, ralucado, Cristina Raluca Vijulie, ralucris.v[at]gmail[dot]com
2019/02/23, gedimitr, Gerasimos Dimitriadis, gedimitr@gmail.com
2019/03/13, base698, Justin Thomas, justin.thomas1@gmail.com
@ -296,6 +293,7 @@ YYYY/MM/DD, github id, Full name, email
2021/02/28, Dante-Broggi, Dante Broggi, 34220985+Dante-Broggi@users.noreply.github.com
2021/03/02, hackeris
2021/03/03, xTachyon, Damian Andrei, xTachyon@users.noreply.github.com
2021/03/22, 100mango, Fangqiu Fang, 100mango@gmail.com
2021/04/07, b1f6c1c4, Jinzheng Tu, b1f6c1c4@gmail.com
2021/04/17, jaggerjo, Josua Jäger, mail@jaggerjo.com
2021/04/24, bigerl, Alexander Bigerl, alexander [äät] bigerl [pkt] eu
@ -305,9 +303,11 @@ YYYY/MM/DD, github id, Full name, email
2021/05/04, joakker, Joaquín León, joaquinandresleon108@gmail.com
2021/05/06, renancaraujo, Renan C. Araújo, renancaraujo@users.noreply.github.com
2021/05/06, canastro, Ricardo Canastro, ricardocanastro@users.noreply.github.com
2021/06/19, abe149, Abe Skolnik, abe 149 at gmail . com
2021/07/01, appel1, Johan Appelgren, johan.appelgren@gmail.com
2021/07/01, marcauberer, Marc Auberer, marc.auberer@chillibits.com
2021/07/14, renzhentaxibaerde, Renzhentaxi Baerde, renzhentaxibaerde@gmail.com
2021/07/21, skittlepox, Benjamin Spiegel, bspiegel@cs.brown.edu
2021/07/29, ksyx, Qixing Xue, qixingxue@outlook.com
2021/07/29, rachidlamouri, Rachid Lamouri, rachidlamouri@gmail.com
2021/08/02, minjoosur, Minjoo Sur, msur@salesforce.com
@ -316,6 +316,15 @@ YYYY/MM/DD, github id, Full name, email
2021/08/25, XenoAmess, Jin Xu, xenoamess@gmail.com
2021/09/08, jmcken8, Joel McKenzie, joel.b.mckenzie@gmail.com
2021/09/23, skalt, Steven Kalt, kalt.steven@gmail.com
2021/09/26, idraper, Isaac Draper, idraper@byu.edu
2021/10/10, tools4origins, Erwan Guyomarc'h, contact@erwan-guyomarch.fr
2021/10/19, jcking, Justin King, jcking@google.com
2021/10/31, skef, Skef Iterum, github@skef.org
2021/10/31, skef, Skef Iterum, github@skef.org
2021/10/31, hlstwizard, h.l.s.t@163.com
2021/11/30, bollwyvl, Nick Bollweg, bollwyvl@users.noreply.github.com
2021/12/03, eneko, Eneko Alonso, eneko.alonso@gmail.com
2021/12/16, Ketler13, Oleksandr Martyshchenko, oleksandr.martyshchenko@gmail.com
2021/12/25, Tinker1024, Tinker1024, tinker@huawei.com
2021/12/31, Biswa96, Biswapriyo Nath, nathbappai@gmail.com
2022/03/07, chenquan, chenquan, chenquan.dev@gmail.com
2022/03/15, hzeller, Henner Zeller, h.zeller@acm.org

16
pom.xml
View File

@ -13,7 +13,7 @@
</parent>
<groupId>org.antlr</groupId>
<artifactId>antlr4-master</artifactId>
<version>4.9.4-SNAPSHOT</version>
<version>4.10-SNAPSHOT</version>
<packaging>pom</packaging>
<name>ANTLR 4</name>
@ -25,7 +25,11 @@
<url>http://www.antlr.org</url>
</organization>
<licenses>
<prerequisites>
<maven>3.8</maven>
</prerequisites>
<licenses>
<license>
<name>The BSD License</name>
<url>http://www.antlr.org/license.html</url>
@ -87,8 +91,6 @@
<module>tool</module>
<module>antlr4-maven-plugin</module>
<module>tool-testsuite</module>
<module>runtime-testsuite/annotations</module>
<module>runtime-testsuite/processors</module>
<module>runtime-testsuite</module>
</modules>
@ -96,8 +98,8 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<antlr.testinprocess>true</antlr.testinprocess>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<mailingLists>
@ -133,7 +135,7 @@
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.0.0</version>
<version>3.1.0</version>
<configuration>
<filesets>
<fileset>

51
runtime-testsuite/annotations/pom.xml
View File

@ -1,51 +0,0 @@
<!--
~ Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
~ Use of this file is governed by the BSD 3-clause license that
~ can be found in the LICENSE.txt file in the project root.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.antlr</groupId>
<artifactId>antlr4-master</artifactId>
<version>4.9.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>antlr4-runtime-test-annotations</artifactId>
<name>ANTLR 4 Runtime Test Annotations</name>
<description>The ANTLR 4 Runtime</description>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<id>bundle-manifest</id>
<phase>process-classes</phase>
<configuration>
<instructions>
<Bundle-SymbolicName>org.antlr.antlr4-runtime-osgi</Bundle-SymbolicName>
<Implementation-Title>ANTLR 4 Runtime</Implementation-Title>
<Implementation-Vendor>ANTLR</Implementation-Vendor>
<Implementation-Vendor-Id>org.antlr</Implementation-Vendor-Id>
<Implementation-Version>${project.version}</Implementation-Version>
</instructions>
</configuration>
<goals>
<goal>manifest</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
</plugin>
</plugins>
</build>
</project>

20
runtime-testsuite/annotations/src/org/antlr/v4/test/runtime/CommentHasStringValue.java
View File

@ -1,20 +0,0 @@
/*
* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
* Use of this file is governed by the BSD 3-clause license that
* can be found in the LICENSE.txt file in the project root.
*/
package org.antlr.v4.test.runtime;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
/** This is just a tag that indicates the javadoc comment has a multi-line string */
@Retention(RetentionPolicy.SOURCE)
@Target({ElementType.FIELD, ElementType.METHOD})
@Inherited
public @interface CommentHasStringValue {
}

42
runtime-testsuite/pom.xml
View File

@ -10,14 +10,14 @@
<parent>
<groupId>org.antlr</groupId>
<artifactId>antlr4-master</artifactId>
<version>4.9.4-SNAPSHOT</version>
<version>4.10-SNAPSHOT</version>
</parent>
<artifactId>antlr4-runtime-testsuite</artifactId>
<name>ANTLR 4 Runtime Tests (2nd generation)</name>
<name>ANTLR 4 Runtime Tests (3rd generation)</name>
<description>A collection of tests for ANTLR 4 Runtime libraries.</description>
<prerequisites>
<maven>3.0</maven>
<maven>3.8</maven>
</prerequisites>
<inceptionYear>2009</inceptionYear>
@ -41,34 +41,22 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime-test-annotations</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime-test-annotation-processors</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>javax.json</artifactId>
<version>1.0.4</version>
<version>1.1.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.openjdk.jol</groupId>
<artifactId>jol-core</artifactId>
<version>0.8</version>
<version>0.16</version>
</dependency>
</dependencies>
@ -91,13 +79,14 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<version>2.22.0</version>
<configuration>
<!-- SUREFIRE-951: file.encoding cannot be set via systemPropertyVariables -->
<argLine>-Dfile.encoding=UTF-8</argLine>
<includes>
<include>**/csharp/Test*.java</include>
<include>**/java/Test*.java</include>
<include>**/java/api/Test*.java</include>
<include>**/go/Test*.java</include>
<include>**/javascript/Test*.java</include>
<include>**/python2/Test*.java</include>
@ -106,12 +95,12 @@
<include>**/dart/Test*.java</include>
<include>${antlr.tests.swift}</include>
</includes>
</configuration>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<version>3.2.0</version>
<executions>
<execution>
<goals>
@ -138,7 +127,16 @@
</execution>
</executions>
</plugin>
</plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<release>8</release>
<source>9</source>
<target>9</target>
</configuration>
</plugin>
</plugins>
</build>
<profiles>

54
runtime-testsuite/processors/pom.xml
View File

@ -1,54 +0,0 @@
<!--
~ Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
~ Use of this file is governed by the BSD 3-clause license that
~ can be found in the LICENSE.txt file in the project root.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.antlr</groupId>
<artifactId>antlr4-master</artifactId>
<version>4.9.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>antlr4-runtime-test-annotation-processors</artifactId>
<name>ANTLR 4 Runtime Test Processors</name>
<description>The ANTLR 4 Runtime</description>
<dependencies>
<dependency>
<groupId>com.github.olivergondza</groupId>
<artifactId>maven-jdk-tools-wrapper</artifactId>
<version>0.1</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime-test-annotations</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src</sourceDirectory>
<resources>
<resource>
<directory>resources</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<verbose>true</verbose>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<compilerArgument>
-proc:none
</compilerArgument>
</configuration>
</plugin>
</plugins>
</build>
</project>

7
runtime-testsuite/processors/resources/META-INF/services/javax.annotation.processing.Processor
View File

@ -1,7 +0,0 @@
#
# Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
# Use of this file is governed by the BSD 3-clause license that
# can be found in the LICENSE.txt file in the project root.
#
org.antlr.v4.test.runtime.CommentHasStringValueProcessor

94
runtime-testsuite/processors/src/org/antlr/v4/test/runtime/CommentHasStringValueProcessor.java
View File

@ -1,94 +0,0 @@
/*
* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
* Use of this file is governed by the BSD 3-clause license that
* can be found in the LICENSE.txt file in the project root.
*/
package org.antlr.v4.test.runtime;
import com.sun.tools.javac.main.JavaCompiler;
import com.sun.tools.javac.model.JavacElements;
import com.sun.tools.javac.tree.JCTree;
import com.sun.tools.javac.tree.TreeMaker;
import com.sun.tools.javac.util.Context;
import com.sun.tools.javac.util.List;
import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import java.lang.reflect.Field;
import java.util.Set;
/**
I think I figured out how to use annotation processors in maven. It's
more or less automatic and you don't even need to tell maven, with one minor
exception. The idea is to create a project for the annotation and another
for the annotation processor. Then, a project that uses the annotation
can simply set up the dependency on the other projects. You have to turn
off processing, -proc:none on the processor project itself but other than
that, java 6+ more or less tries to apply any processors it finds during
compilation. maven just works.
Also you need a META-INF/services/javax.annotation.processing.Processor file
with "org.antlr.v4.test.runtime.CommentHasStringValueProcessor" in it.
*/
@SupportedAnnotationTypes({"org.antlr.v4.test.runtime.CommentHasStringValue"})
@SupportedSourceVersion(SourceVersion.RELEASE_7)
public class CommentHasStringValueProcessor extends AbstractProcessor {
protected JavacElements utilities;
protected TreeMaker treeMaker;
@Override
public synchronized void init(ProcessingEnvironment processingEnv) {
super.init(processingEnv);
// Messager messager = processingEnv.getMessager();
// messager.printMessage(Diagnostic.Kind.NOTE, "WOW INIT--------------------");
utilities = (JavacElements)processingEnv.getElementUtils();
treeMaker = TreeMaker.instance(extractContext(utilities));
}
private static Context extractContext(JavacElements utilities) {
try {
Field compilerField = JavacElements.class.getDeclaredField("javaCompiler");
compilerField.setAccessible(true);
JavaCompiler compiler = (JavaCompiler)compilerField.get(utilities);
Field contextField = JavaCompiler.class.getDeclaredField("context");
contextField.setAccessible(true);
return (Context)contextField.get(compiler);
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
@Override
public boolean process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) {
// Messager messager = processingEnv.getMessager();
// messager.printMessage(Diagnostic.Kind.NOTE, "PROCESS--------------------");
Set<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(CommentHasStringValue.class);
for (Element annotatedElement : annotatedElements) {
String docComment = utilities.getDocComment(annotatedElement);
JCTree.JCLiteral literal = treeMaker.Literal(docComment!=null ? docComment : "");
JCTree elementTree = utilities.getTree(annotatedElement);
if ( elementTree instanceof JCTree.JCVariableDecl ) {
((JCTree.JCVariableDecl)elementTree).init = literal;
}
else if ( elementTree instanceof JCTree.JCMethodDecl ) {
JCTree.JCStatement[] statements = new JCTree.JCStatement[1];
statements[0] = treeMaker.Return(literal);
((JCTree.JCMethodDecl)elementTree).body = treeMaker.Block(0, List.from(statements));
}
}
return true;
}
@Override
public SourceVersion getSupportedSourceVersion() {
return SourceVersion.latestSupported();
}
}

4003
runtime-testsuite/resources/org/antlr/v4/test/runtime/LargeLexer.g4
View File

File diff suppressed because it is too large Load Diff

24
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeLexers/LexerDelegatorInvokesDelegateRule.txt Normal file
View File

@ -0,0 +1,24 @@
[type]
CompositeLexer
[grammar]
lexer grammar M;
import S;
B : 'b';
WS : (' '|'\n') -> skip ;
[slaveGrammar]
lexer grammar S;
A : 'a' {<writeln("\"S.A\"")>};
C : 'c' ;
[input]
abc
[output]
S.A
[@0,0:0='a',<3>,1:0]
[@1,1:1='b',<1>,1:1]
[@2,2:2='c',<4>,1:2]
[@3,3:2='<EOF>',<-1>,1:3]

22
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeLexers/LexerDelegatorRuleOverridesDelegate.txt Normal file
View File

@ -0,0 +1,22 @@
[type]
CompositeLexer
[grammar]
lexer grammar M;
import S;
A : 'a' B {<writeln("\"M.A\"")>} ;
WS : (' '|'\n') -> skip ;
[slaveGrammar]
lexer grammar S;
A : 'a' {<writeln("\"S.A\"")>} ;
B : 'b' {<writeln("\"S.B\"")>} ;
[input]
ab
[output]
M.A
[@0,0:1='ab',<1>,1:0]
[@1,2:1='<EOF>',<-1>,1:2]

23
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/BringInLiteralsFromDelegate.txt Normal file
View File

@ -0,0 +1,23 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
s : a ;
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
a : '=' 'a' {<write("\"S.a\"")>};
[start]
s
[input]
=a
[output]
"""S.a
"""

25
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/CombinedImportsCombined.txt Normal file
View File

@ -0,0 +1,25 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
s : x INT;
[slaveGrammar]
parser grammar S;
tokens { A, B, C }
x : 'x' INT {<writeln("\"S.x\"")>};
INT : '0'..'9'+ ;
WS : (' '|'\n') -> skip ;
[start]
s
[input]
x 34 9
[output]
"""S.x
"""

44
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatesSeeSameTokenType.txt Normal file
View File

@ -0,0 +1,44 @@
[notes]
The lexer will create rules to match letters a, b, c.
The associated token types A, B, C must have the same value
and all import'd parsers. Since ANTLR regenerates all imports
for use with the delegator M, it can generate the same token type
mapping in each parser:
public static final int C=6;
public static final int EOF=-1;
public static final int B=5;
public static final int WS=7;
public static final int A=4;
[type]
CompositeParser
[grammar]
grammar M;
import S,T;
s : x y ; // matches AA, which should be 'aa'
B : 'b' ; // another order: B, A, C
A : 'a' ;
C : 'c' ;
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar T;
tokens { C, B, A } // reverse order
y : A {<writeln("\"T.y\"")>};
[slaveGrammar]
parser grammar S;
tokens { A, B, C }
x : A {<writeln("\"S.x\"")>};
[start]
s
[input]
aa
[output]
S.x
T.y

26
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorAccessesDelegateMembers.txt Normal file
View File

@ -0,0 +1,26 @@
[type]
CompositeParser
[grammar]
grammar M; // uses no rules from the import
import S;
s : 'b' {<Invoke_foo()>} ; // gS is import pointer
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
@parser::members {
<Declare_foo()>
}
a : B;
[start]
s
[input]
b
[output]
"""foo
"""

24
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorInvokesDelegateRule.txt Normal file
View File

@ -0,0 +1,24 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
s : a ;
B : 'b' ; // defines B from inherited token space
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
a : B {<writeln("\"S.a\"")>};
[start]
s
[input]
b
[output]
"""S.a
"""

24
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorInvokesDelegateRuleWithArgs.txt Normal file
View File

@ -0,0 +1,24 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
s : label=a[3] {<writeln("$label.y")>} ;
B : 'b' ; // defines B from inherited token space
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
a[int x] returns [int y] : B {<write("\"S.a\"")>} {$y=1000;} ;
[start]
s
[input]
b
[output]
"""S.a1000
"""

24
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorInvokesDelegateRuleWithReturnStruct.txt Normal file
View File

@ -0,0 +1,24 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
s : a {<write("$a.text")>} ;
B : 'b' ; // defines B from inherited token space
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
a : B {<write("\"S.a\"")>} ;
[start]
s
[input]
b
[output]
"""S.ab
"""

29
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorInvokesFirstVersionOfDelegateRule.txt Normal file
View File

@ -0,0 +1,29 @@
[type]
CompositeParser
[grammar]
grammar M;
import S,T;
s : a ;
B : 'b' ; // defines B from inherited token space
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar T;
a : B {<writeln("\"T.a\"")>};<! hidden by S.a !>
[slaveGrammar]
parser grammar S;
a : b {<writeln("\"S.a\"")>};
b : B;
[start]
s
[input]
b
[output]
"""S.a
"""

24
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorRuleOverridesDelegate.txt Normal file
View File

@ -0,0 +1,24 @@
[type]
CompositeParser
[grammar]
grammar M;
import S;
b : 'b'|'c';
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar S;
a : b {<write("\"S.a\"")>};
b : B ;
[start]
a
[input]
c
[output]
"""S.a
"""

29
runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors/CompositeParsers/DelegatorRuleOverridesDelegates.txt Normal file
View File

@ -0,0 +1,29 @@
[type]
CompositeParser
[grammar]
grammar M;
import S, T;
b : 'b'|'c' {<writeln("\"M.b\"")>}|B|A;
WS : (' '|'\n') -> skip ;
[slaveGrammar]
parser grammar T;
tokens { A }
b : 'b' {<writeln("\"T.b\"")>};
[slaveGrammar]
parser grammar S;
a : b {<writeln("\"S.a\"")>};
b : 'b' ;
[start]
a
[input]
c
[output]
M.b
S.a

Some files were not shown because too many files have changed in this diff Show More