This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
| Summary: | Remove incomplete token ids by providing a PartType enum | ||
|---|---|---|---|
| Product: | editor | Reporter: | Miloslav Metelka <mmetelka> |
| Component: | Lexer | Assignee: | Miloslav Metelka <mmetelka> |
| Status: | RESOLVED FIXED | ||
| Severity: | blocker | CC: | jjancura, jlahoda, mfukala, vstejskal |
| Priority: | P2 | Keywords: | API_REVIEW_FAST |
| Version: | 6.x | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Issue Type: | TASK | Exception Reporter: | |
| Bug Depends on: | |||
| Bug Blocks: | 87014 | ||
| Attachments: | Diff of the proposed changes | ||
I have added PartType class and modified JavaTokenId to no longer contain
*_INCOMPLETE tokens and updated the depending java code accordingly.
I was selecting between
PartType { COMPLETE, START, INNER, END }
and
PartType { START, INNER, END } and <null> meaning complete token
I have chosen the first solution.
Marking for fasttrack review, diff attached.
Created attachment 39153 [details]
Diff of the proposed changes
The apichanges.xml is talking about PartInfo and Token.partInfo while the added enum is PartType and method Token.partType. Oops, fixed. Thanks, Vita for finding this. IMO the Token.partType() mechanism is generally OK and seems to cover my usecases so I agree with commiting to trunk. I have just two comments: 1) I do not see a reasonable way how to implement the Lexer to recognize that the created token should be PartType.END or INNER since there is no info that it previously returned START token. More context information is needed here. OTOH even if the context was wider and containing the info about previously returned token, how I would recognize that I should return INNER instead of END? I would have to somehow look forward and guess that there will be another token breaking the current one. 2) How the PartType connected with the embedding? More concrete, will the embedding on one token part be applied to the rest of the token parts? Or do I have to handle this manually in the implementation? 3) Just an idea - it would be probably handy to have a TokenSequence method like TokenSequence ts = ...; ts.nextWholeToken(); so the clients of the Lexer API doesn't have to (if they explicitly do not want to) even know about the token parts. Yes, the 1) will be solved as part of issue 87014. Regarding INNER versus END: you would return END only if the token would be complete e.g. "/** ..." would have to be closed by "*/" otherwise you would return INNER. ad 2) and 3) again this should be finalized as part of issue 87014. My current assumption is that there would be a method to return the complete token but the embedding would only be done on the individual sections so that the whole token hierarchy remains to be a tree of tokens in terms of coverage of the lexed text. Looks good. Fixed: Checking in java/editor/nbproject/project.xml; /cvs/java/editor/nbproject/project.xml,v <-- project.xml new revision: 1.25; previous revision: 1.24 done Checking in java/editor/src/org/netbeans/modules/editor/java/Utilities.java; /cvs/java/editor/src/org/netbeans/modules/editor/java/Utilities.java,v <-- Utilities.java new revision: 1.18; previous revision: 1.17 done Checking in java/lexer/manifest.mf; /cvs/java/lexer/manifest.mf,v <-- manifest.mf new revision: 1.5; previous revision: 1.4 done Checking in java/lexer/nbproject/project.xml; /cvs/java/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.5; previous revision: 1.4 done Checking in java/lexer/src/org/netbeans/api/java/lexer/JavaTokenId.java; /cvs/java/lexer/src/org/netbeans/api/java/lexer/JavaTokenId.java,v <-- JavaTokenId.java new revision: 1.8; previous revision: 1.7 done Checking in java/lexer/src/org/netbeans/lib/java/lexer/JavaLexer.java; /cvs/java/lexer/src/org/netbeans/lib/java/lexer/JavaLexer.java,v <-- JavaLexer.java new revision: 1.5; previous revision: 1.4 done Checking in java/lexer/test/unit/data/testfiles/testInput.java.txt.tokens.txt; /cvs/java/lexer/test/unit/data/testfiles/testInput.java.txt.tokens.txt,v <-- testInput.java.txt.tokens.txt new revision: 1.2; previous revision: 1.1 done Checking in java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLanguageTest.java; /cvs/java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLanguageTest.java,v <-- JavaLanguageTest.java new revision: 1.3; previous revision: 1.2 done Checking in java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLexerBatchTest.java; /cvs/java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLexerBatchTest.java,v <-- JavaLexerBatchTest.java new revision: 1.3; previous revision: 1.2 done Checking in java/source/test/unit/src/org/netbeans/api/java/source/gen/WhitespaceIgnoringDiff.java; /cvs/java/source/test/unit/src/org/netbeans/api/java/source/gen/WhitespaceIgnoringDiff.java,v <-- WhitespaceIgnoringDiff.java new revision: 1.4; previous revision: 1.3 done Checking in javadoc/nbproject/project.xml; /cvs/javadoc/nbproject/project.xml,v <-- project.xml new revision: 1.19; previous revision: 1.18 done Checking in javadoc/src/org/netbeans/modules/javadoc/hints/JavadocUtilities.java; /cvs/javadoc/src/org/netbeans/modules/javadoc/hints/JavadocUtilities.java,v <-- JavadocUtilities.java new revision: 1.3; previous revision: 1.2 done Checking in languages/engine/src/org/netbeans/modules/languages/lexer/SLexer.java; /cvs/languages/engine/src/org/netbeans/modules/languages/lexer/SLexer.java,v <-- SLexer.java new revision: 1.16; previous revision: 1.15 done Checking in lexer/manifest.mf; /cvs/lexer/manifest.mf,v <-- manifest.mf new revision: 1.15; previous revision: 1.14 done Checking in lexer/api/apichanges.xml; /cvs/lexer/api/apichanges.xml,v <-- apichanges.xml new revision: 1.13; previous revision: 1.12 done RCS file: /cvs/lexer/src/org/netbeans/api/lexer/PartType.java,v done Checking in lexer/src/org/netbeans/api/lexer/PartType.java; /cvs/lexer/src/org/netbeans/api/lexer/PartType.java,v <-- PartType.java initial revision: 1.1 done Checking in lexer/src/org/netbeans/api/lexer/Token.java; /cvs/lexer/src/org/netbeans/api/lexer/Token.java,v <-- Token.java new revision: 1.8; previous revision: 1.7 done Checking in lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java; /cvs/lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java,v <-- LexerInputOperation.java new revision: 1.7; previous revision: 1.6 done Checking in lexer/src/org/netbeans/lib/lexer/LexerUtilsConstants.java; /cvs/lexer/src/org/netbeans/lib/lexer/LexerUtilsConstants.java,v <-- LexerUtilsConstants.java new revision: 1.8; previous revision: 1.7 done Checking in lexer/src/org/netbeans/lib/lexer/token/AbstractToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/AbstractToken.java,v <-- AbstractToken.java new revision: 1.5; previous revision: 1.4 done RCS file: /cvs/lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java,v done Checking in lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java,v <-- ComplexToken.java initial revision: 1.1 done Checking in lexer/src/org/netbeans/lib/lexer/token/CustomTextToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/CustomTextToken.java,v <-- CustomTextToken.java new revision: 1.5; previous revision: 1.4 done Checking in lexer/src/org/netbeans/lib/lexer/token/DefaultToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/DefaultToken.java,v <-- DefaultToken.java new revision: 1.6; previous revision: 1.5 done Checking in lexer/src/org/netbeans/lib/lexer/token/PreprocessedTextToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/PreprocessedTextToken.java,v <-- PreprocessedTextToken.java new revision: 1.4; previous revision: 1.3 done Removing lexer/src/org/netbeans/lib/lexer/token/PropertyCustomTextToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyCustomTextToken.java,v <-- PropertyCustomTextToken.java new revision: delete; previous revision: 1.3 done Removing lexer/src/org/netbeans/lib/lexer/token/PropertyPreprocessedTextToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyPreprocessedTextToken.java,v <-- PropertyPreprocessedTextToken.java new revision: delete; previous revision: 1.3 done Checking in lexer/src/org/netbeans/lib/lexer/token/PropertyToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyToken.java,v <-- PropertyToken.java new revision: 1.4; previous revision: 1.3 done Checking in lexer/src/org/netbeans/lib/lexer/token/TextToken.java; /cvs/lexer/src/org/netbeans/lib/lexer/token/TextToken.java,v <-- TextToken.java new revision: 1.5; previous revision: 1.4 done Checking in lexer/src/org/netbeans/spi/lexer/TokenFactory.java; /cvs/lexer/src/org/netbeans/spi/lexer/TokenFactory.java,v <-- TokenFactory.java new revision: 1.7; previous revision: 1.6 done Checking in lexer/src/org/netbeans/spi/lexer/TokenPropertyProvider.java; /cvs/lexer/src/org/netbeans/spi/lexer/TokenPropertyProvider.java,v <-- TokenPropertyProvider.java new revision: 1.6; previous revision: 1.5 done Checking in lexer/test/unit/src/org/netbeans/api/lexer/CustomTokenClassTest.java; /cvs/lexer/test/unit/src/org/netbeans/api/lexer/CustomTokenClassTest.java,v <-- CustomTokenClassTest.java new revision: 1.4; previous revision: 1.3 done Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/LexerTestUtilities.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/LexerTestUtilities.java,v <-- LexerTestUtilities.java new revision: 1.9; previous revision: 1.8 done Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpCheck.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpCheck.java,v <-- TokenDumpCheck.java new revision: 1.4; previous revision: 1.3 done Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpLexer.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpLexer.java,v <-- TokenDumpLexer.java new revision: 1.4; previous revision: 1.3 done Checking in web/jspsyntax/lexer/nbproject/project.xml; /cvs/web/jspsyntax/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.8; previous revision: 1.7 done Checking in web/jspsyntax/lexer/src/org/netbeans/lib/jsp/lexer/JspLexer.java; /cvs/web/jspsyntax/lexer/src/org/netbeans/lib/jsp/lexer/JspLexer.java,v <-- JspLexer.java new revision: 1.21; previous revision: 1.20 Just a note: current syntax highlighting ignores PartTypes and there is no way to define a different coloring for particular parts of a token, which in some sense used to be possible with *_INCOMPLETE tokens, but had not been used AFAIK. Do we need this (1) or shall we leave it for later (2) when and if we really need it? As an implication of #2 any colorings registered in the layer files for incomplete tokens (that have now been removed) should also be deleted. Don't forget about the CityLights profile's coloring files. Yes, I was thinking about this before implementation and I assumed that the coloring for the particular incomplete token-ids will usually be the same like the coloring for their corresponding complete counterpart. I have also thought that we could eventually make a generic algorithm that would inspect the incomplete tokens and make e.g. a composition of their color with red color to point out that their are incomplete (or it could e.g. make a curly underline). But this needs to be done after completion of issue 87014 because some of these incomplete tokens will be result of lexing multiple embedded sections at once. I'll fix the colorings. I have checked the colorings and they are only per-category e.g. for "string" so they are unafected by this change. |
This is a request to remove incomplete token ids (e.g. JavaTokenId.BLOCK_COMMENT_INCOMPLETE) that clutter the language definition. Instead there would be an enum TokenPartType { COMPLETE, START, INNER, END }. Token.partType() would return a non-null enum value. For the most typical situation when the token is complete the COMPLETE would be returned. For the present incomplete token ids the START would be used. The INNER and END would cover the requirements of issue 87014. The TokenFactory would be enhanced accordingly to allow the creation of the token parts.