Skip to content

Template Toolkit support: 105/106 tests passing (99%)#441

Open
fglock wants to merge 14 commits intomasterfrom
feature/fix-template-toolkit
Open

Template Toolkit support: 105/106 tests passing (99%)#441
fglock wants to merge 14 commits intomasterfrom
feature/fix-template-toolkit

Conversation

@fglock
Copy link
Copy Markdown
Owner

@fglock fglock commented Apr 5, 2026 •
edited
Loading

Summary

Fixes multiple bugs uncovered by the Template Toolkit 3.102 test suite, bringing
test pass rate from 87/106 to 105/106 tests passing (2882/2884 subtests, 99.9% subtest pass rate).

The only remaining failure is leak.t (2 subtests) which requires DESTROY support -- a known PerlOnJava limitation.

Fixes in this branch (9 fixes)

  1. JAR shim preservation -- MakeMaker no longer overwrites jar:PERL5LIB shims when installing CPAN modules (ExtUtils/MakeMaker.pm)

  2. use constant / our clash -- use constant ERROR => 2; our $ERROR = '' no longer throws "Modification of a read-only value" (RuntimeStashEntry.java)

  3. Interpreter our variable binding -- our declarations in interpreter-fallback code now correctly bind to package globals (BytecodeCompiler.java)

  4. XSLoader @isa fallback -- XS modules with pure-Perl parents via @isa now load successfully even without Java XS classes (XSLoader.java)

  5. Error location attribution -- Improved file/line reporting in compiled templates and interpreter frames (PerlCompilerException.java, ByteCodeSourceMapper.java, WarnDie.java)

  6. Compiled template loading -- Fixed utime, error location in .ttc files, filehandle context, $. counting in slurp/paragraph mode (multiple files)

  7. XSLoader jar: shim overrides -- After @isa fallback, XSLoader now loads method overrides from jar:PERL5LIB shims. Also fixed @_ in eval'd sub definitions. (XSLoader.java, EvalStringHandler.java, Template/Stash/XS.pm)

  8. use bytes length for Latin-1 -- lengthBytes() now returns character count for strings with all chars <= 0xFF, fixing BOM detection in Template::Provider (StringOperators.java)

  9. Scoped packageExistsCache sub entries -- sub error in Template::Base no longer prevents error from being used as a class name in indirect method syntax from other packages (SubroutineParser.java)

Other fixes (from earlier commits)

  • Regex $| interpolation inside alternation
  • Circular blessed reference stringification
  • CvSTASH for anonymous subs (AUTOLOAD package)
  • \r handling in lexer (Template Toolkit string literals)
  • DATA section UTF-8 decoding
  • Operation context in "isn't numeric" warnings

Test results

Metric Before After
Test programs passing 87/106 105/106
Subtests passing ~2300/2884 2882/2884
Pass rate ~82% 99.9%

Remaining failure

  • leak.t (2/11 subtests) -- Tests 7, 11 verify DESTROY is called. PerlOnJava does not implement DESTROY (JVM tracing GC handles circular references natively). This is a known limitation.

Note on parallel test ordering

When running ./jcpan -j 8 -t Template, the compile tests (compile2.t, compile3.t, compile5.t) may spuriously fail because they depend on compile1.t running first. This is a test-harness ordering issue, not a PerlOnJava bug. All compile tests pass when run sequentially.

Test plan

  • All unit tests pass (make)
  • TT test suite: 105/106 programs, 2882/2884 subtests
  • No regressions in existing tests
  • Design doc updated: dev/modules/template.md

Generated with Devin

@fglock fglock changed the title Template Toolkit support: 97/106 tests passing (97.7% subtest pass rate) Template Toolkit support: 105/106 tests passing (99%) Apr 5, 2026
fglock and others added 14 commits April 5, 2026 22:00
@fglock @devin-ai-integration
Template Toolkit support: 87/106 tests passing (82%)
21ed18b 
Four fixes to support Template Toolkit 3.102:

1. use constant / our $VAR clash (RuntimeStashEntry.java)
   - Stash assignment of scalar ref no longer overwrites the scalar
     variable slot; only creates the constant subroutine (&name)

2. Interpreter fallback `our` variable binding (BytecodeCompiler.java)
   - When BEGIN blocks fall back to interpreter, `our` declarations
     now emit LOAD_GLOBAL_SCALAR/ARRAY/HASH even when hasVariable()
     returns true, rebinding registers to package globals
   - Fixes Template::Grammar's ~6000-line BEGIN block

3. XSLoader @isa fallback (XSLoader.java)
   - When no Java XS class exists but the module's @isa already has
     a pure-Perl parent, return success instead of dying
   - Enables CPAN XS .pm files to load naturally through inheritance

4. MakeMaker JAR shim preservation (ExtUtils/MakeMaker.pm)
   - Skip installing CPAN .pm files that have a PerlOnJava shim in
     the JAR, preventing XSLoader-based files from shadowing shims

See dev/modules/template.md for detailed analysis of remaining failures.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: @_ strict vars error in nested eval sub definitions
ba97247 
EvalStringHandler.evalStringList created a fresh empty ScopedSymbolTable
without the standard variables (this, @_, wantarray). This caused named
subs parsed inside interpreter-path eval to get an empty filteredSnapshot,
making @_ invisible during lazy JVM compilation and triggering a spurious
strict vars error for code like: eval q{ eval q{sub foo { shift }} }

The fix adds enterScope() and the three standard variables to match
PerlLanguageProvider and evalStringWithInterpreter initialization.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
feat: add operation context to "isn't numeric" warning messages
aa4040d 
Thread operation context (e.g., "in addition (+)", "in numeric lt (<)")
through getNumber() to NumberParser.parseNumber() so that the warning
"Argument X isn't numeric" includes the operation name, matching Perl 5.

Changes:
- NumberParser: add parseNumber(RuntimeScalar, String) overload
- RuntimeScalar: add getNumber(String) overload for string types
- MathOperators: pass operation name in add, subtract, multiply, divide,
  abs, and int methods
- CompareOperators: pass operation name in all numeric comparison methods

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: Template Toolkit test fixes - regex, circular bless, CvSTASH, DA…
0d8ac5c 
…TA UTF-8

- Fix regex $| interpolation in patterns (StringSegmentParser.java)
- Fix circular blessed reference stringification (RuntimeScalar.java)
- Add CvSTASH support for anonymous subs (EmitSubroutine.java, RuntimeCode.java, InheritanceResolver.java)
- Fix \r handling in lexer for eval'd code (Lexer.java, FileUtils.java)
- Add DATA section UTF-8 decoding when use utf8 active (DataSection.java)
- Improve warn/die location reporting for interpreter frames (WarnDie.java)

All 8 previously-failing TT tests now pass:
  t/html.t (18/18), t/url.t (23/23), t/fileline.t (11/11),
  t/context.t (54/54), t/meta.t (3/3), t/tiedhash.t (51/51),
  t/proc.t (7/7), t/parser.t (37/37)

Generated with Devin (https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
Fix $1 capture NPE in numeric context and DATA section UTF-8 encoding
205c393 
ScalarSpecialVariable was missing a getNumber(String) override, causing
$1 + 0, $1 > 0, etc. to crash with "Cannot invoke getDoubleRef() because
runtimeScalar.value is null". The call path went through RuntimeScalar's
getNumber(String) which fell through to Overload.numify() where the PROXY
type's null value field caused the NPE. This fix adds the missing override,
fixing 34 tests in Template Toolkit's vmethods/replace.t.

DATA section UTF-8 handling was also incorrect: raw bytes were decoded as
UTF-8 at extraction time, but ScalarBackedIO.doRead() converts the backing
string to bytes using ISO-8859-1, destroying characters > 0xFF. The fix
stores DATA content as raw bytes always and applies a :utf8 IO layer to
the DATA handle when `use utf8` is active, matching Perl 5 behavior where
the encoding layer handles decoding at read time. This fixes 2 tests in
Template Toolkit's stash-xs-unicode.t.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: use current execution location for error messages instead of cal…
833b32c 
…ler(0)

PerlCompilerException.buildErrorMessage() was using caller(0) which returns
the CALL SITE (where the current sub was called from), not the CURRENT
EXECUTION LOCATION (where the error actually occurs). This caused wrong
file/line in error messages when code with #line directives was called
from a different file — e.g. "at outer.tt line 1" instead of "at inner.tt
line 1" for Template Toolkit compiled templates.

The fix checks the interpreter's current PC first (which correctly tracks
#line directives via ErrorMessageUtil), then falls back to JVM stack
scanning for compiled Perl code. This matches Perl 5's COP-based error
location behavior.

Fixes TT compile1.t (17/18 -> 18/18) and compile4.t (12/13 -> 13/13).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: error location, filehandle context, and dollar-dot in slurp/para…
8a3fee4 
…graph mode

- PerlCompilerException: detect interpreter vs JVM innermost context
  by scanning stack for bytecode.* frames before anon classes. This
  fixes error location for both cached (.ttc) and non-cached templates.

- ByteCodeSourceMapper: use getSourceLocationAccurate() to honor #line
  directives in JVM-compiled code.

- WarnDie: implement Perl 5 filehandle context suffix for error messages
  using RuntimeIO.globName. Uses line only when dollar-slash is newline,
  chunk for everything else (undef, empty, custom separator).

- Readline: fix dollar-dot counting in slurp mode (increment by 1, not
  per line) and paragraph mode (increment per paragraph, not per line).

- RuntimeIO.close(): reset currentLineNumber to 0, matching Perl 5
  behavior where dollar-dot becomes 0 after closing a filehandle.

- UtimeOperator: fix utime to properly handle timestamp arguments.

Fixes TT compile1.t (18/18), compile4.t (13/13), compile5.t (14/14).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
docs: update template.md with compile1/4/5 fix progress
239b81b 
Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: XSLoader loads jar: shim overrides after @isa fallback
e31a352 
When XSLoader::load falls back to @isa inheritance (no Java XS class),
the CPAN-installed .pm is already loaded. Our jar: PERL5LIB may have a
shim with bug fixes that should be applied on top. XSLoader now reads
and evals the jar: version to install method overrides.

This fixes tiedhash.t (51/51): Template::Stash::XS _assign() used
$root->{$item} (hash deref) on array refs when $default was true,
causing "Not a HASH reference" errors. The jar: shim corrects this
to $root->[$item].

Also fixes EvalStringHandler's simple evalString overload to include
@_ in its symbol table, matching the evalStringList overload. Without
this, subs defined in eval'd code failed strict vars checks for @_.

Template Toolkit: 103/106 passing (97%), up from 90/106.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: use bytes length returns Latin-1 count for chars <= 0xFF
2afb224 
In Perl 5, strings with all characters <= 0xFF are stored internally
as Latin-1 (1 byte per char). Under 'use bytes', length() returns the
internal byte count, which equals the character count for Latin-1.

PerlOnJava's lengthBytes() always converted to UTF-8, giving 2 bytes
per char for 0x80-0xFF range. This broke Template::Provider's BOM
detection, which uses 'use bytes; length($bom)' to get BOM pattern
sizes.

Fix: lengthBytes() now returns str.length() when all chars <= 0xFF,
and only uses UTF-8 byte count for strings with chars > 0xFF.

Template Toolkit unicode.t: 0/20 -> 20/20 (all BOM formats working).
Template Toolkit overall: 104/106 passing (98%).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: scope packageExistsCache sub entries to their declaring package
73b29ca 
When `sub error` was defined in e.g. Template::Base, the parser stored
the bare name "error" in packageExistsCache as false (non-package).
This incorrectly prevented `error` from being used as a class name
in indirect method syntax from other packages — for example,
`parse error` in main should still parse as `error->parse()`.

The fix qualifies sub names with their declaring package before
storing in the cache (e.g., "Template::Base::error" instead of
"error"), and the lookup now checks the qualified name in the
current package context.

This fixes evalperl.t test 11 in Template Toolkit (TT 105/106).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
docs: update template.md — TT 105/106 (99% pass rate)
3a3e966 
Fix 9 scoped packageExistsCache sub entries to their declaring
package, fixing evalperl.t test 11. Only leak.t remains failing
(DESTROY not implemented).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
docs: expand next steps in template.md
966d4e6 
- Branch is ready for merge — all actionable failures fixed
- Document jcpan parallel test ordering note for compile tests
- Clarify leak.t limitation (DESTROY not implemented)

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix: resolve op/length.t and io/open.t regressions
1d9f59f 
op/length.t (40->38->40): Fix lengthBytes to use BYTE_STRING/STRING
type distinction instead of character range heuristic. BYTE_STRING
(Latin-1, SvUTF8=0 equivalent) returns char count; STRING (UTF-8
upgraded, SvUTF8=1 equivalent) returns UTF-8 byte count. This
correctly handles pack("U", 0xFF) returning 2 bytes under use bytes
while keeping \x{ef} at 1 byte.

io/open.t (187->185->187): Track readline handle names for error
messages via AST annotation. The parser annotates readline nodes
with the source variable name (e.g., "$f", "STDIN"). The JVM
backend sets RuntimeIO.lastReadlineHandleName before calling
readline. WarnDie.findFilehandleName uses this as fallback when
the handle globName is null (lexical filehandles).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock fglock force-pushed the feature/fix-template-toolkit branch from 9bbc1e7 to 1d9f59f Compare April 5, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fglock