From Newsgroup: comp.lang.forth

Let's say I convert a string into an anonymous function:

." :noname 3 3 + ;" EVALUATE

This leaves an xt on the stack. The compiled code lives in the
dictionary, right? So if I dynamically create a lot of such xt's for
temporary use, that's a memory leak? If each one is supposed to be used
and released immediately after creation, FORGET could rewind the
dictionary to the right place, but maybe the required lifetimes are more complex.

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

This would be the garbage collected heap using Anton's GC, if it
matters.

Why to want this? Muhahaha.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> writes:

." :noname 3 3 + ;" EVALUATE

Oops meant S" not ." obviously. Sorry.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> writes:

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

I wonder if it suffices to temporarily set DP (the gforth dictionary
pointer, not documented in the gforth manual) to point into the
allocated memory. To allocate the correct sized block, maybe compile
twice. Something like (untested):

: compile-str { a u -- xt newloc }
HERE { old-here }
a u EVALUATE ( compile into dictionary )
DROP \ don't need the xt this time
ALIGN HERE old-here - { size }
size ALLOC { newloc }
newloc DP ! \ now HERE points to the heap
a u EVALUATE ( xt ) \ re-do the compilation
old-here DP ! \ restore dictionary
newloc ( xt newloc)
;

The xt and newloc both have to be kept around, so the xt can be executed
and so the GC can find the allocated block, but both can be copied into
some other ALLOC'd object which the GC will scan.

Alternatively, if the compiler works in a way that xt and newloc are
known to be the same address, there's no need to remember both.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> writes:

Paul Rubin <no.email@nospam.invalid> writes:

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

I wonder if it suffices to temporarily set DP (the gforth dictionary
pointer, not documented in the gforth manual) to point into the
allocated memory.

There are a few difficulties:

1) ALLOT checks whether the resulting DP is in the dictionary, and
reports an error if not. In recent development versions, "," calls
SMALL-ALLOT, which does not do that check (instead, guard pages
detect dictionary overflow), but, e.g., SLITERAL (called by S")
calls ALLOT. But as long as you use a recent development version
and avoid words that call ALLOT, this should work.

2) If Gforth was built well, by default it does not just generate the
threaded code at HERE (where DP points to), but also native code
elsewhere. That native code storage is not controlled by the
garbage collector or free, and it is designed for a LIFO
reclamation discipline (and actually that, i.e., MARKER, already
cost far more development time than the benefits gained from
MARKER). So if you want to avoid memory leaks, you will want to
avoid the dynamic native-code generation by starting gforth (or
gforth-fast) with the option "--no-dynamic", or you just call
gforth-itc.

All these options normally cost performance, but given that the
native-code generation costs time, in your case you might even come
out ahead by disabling native-code generation, depending on your
ratio of generation vs. execution.

: compile-str { a u -- xt newloc }
HERE { old-here }
a u EVALUATE ( compile into dictionary )
DROP \ don't need the xt this time
ALIGN HERE old-here - { size }
size ALLOC { newloc }
newloc DP ! \ now HERE points to the heap
a u EVALUATE ( xt ) \ re-do the compilation
old-here DP ! \ restore dictionary
newloc ( xt newloc)
;

The xt and newloc both have to be kept around, so the xt can be executed
and so the GC can find the allocated block, but both can be copied into
some other ALLOC'd object which the GC will scan.

Alternatively, if the compiler works in a way that xt and newloc are
known to be the same address, there's no need to remember both.

In recent Gforth the xt points at the body, and there is the header
before that, so newloc and xt are not the same. You can configure my
garbage collector to treat internal pointers as keeping the allocation
alive, then you only need to keep the xt around.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2026 CFP: http://www.euroforth.org/ef26/cfp.html
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Thanks for the response. After giving it some thought I believe I can
stop worrying about this problem and just live with the memory leak.
The one situation where it can become an issue is interactive, so I can
just restart gforth if the memory gets full. I do want to use native
code generation or whatever other features are in gforth.

You can configure my garbage collector to treat internal pointers as
keeping the allocation alive, then you only need to keep the xt around.

Ah, this is nice. I may have missed that in the html doc somehow.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> writes:

You can configure my garbage collector to treat internal pointers as
keeping the allocation alive, then you only need to keep the xt around.

Ah, this is nice. I may have missed that in the html doc somehow.

That's probably because it does not exist in released versions. I
have found code where I started to work on it in 2018, but apparently
I have not finished that work.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2026 CFP: http://www.euroforth.org/ef26/cfp.html
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:

2) If Gforth was built well, by default it does not just generate the
threaded code at HERE (where DP points to), but also native code
elsewhere. That native code storage is not controlled by the
garbage collector or free, and it is designed for a LIFO
reclamation discipline (and actually that, i.e., MARKER, already
cost far more development time than the benefits gained from
MARKER). So if you want to avoid memory leaks, you will want to
avoid the dynamic native-code generation by starting gforth (or
gforth-fast) with the option "--no-dynamic", or you just call
gforth-itc.

All these options normally cost performance, but given that the
native-code generation costs time, in your case you might even come
out ahead by disabling native-code generation, depending on your
ratio of generation vs. execution.

This got me into thinking that one could make Gforth more flexible:
Generate native code for the threaded code that is compiled into the dictionary, but also support generating threaded code elsewhere that
gets no native code.

Now if you have a sequence of primitives to generate in your
heap-allocated threaded code, such as "+ C@", this would result in two
NEXTs (threaded-code dispatches). You could try to put it into a
colon definition:

: +C@ + C@ ;

an that reduces the NEXTs to one, but incurs the overhead of saving
the instruction pointer (IP) to the return stack and restoring it from
there.

However, if all the words in the colon definition are primitives and
none of them needs the instruction pointer (this requires that they
are all dynamically compiled to native code with the IP-update
optimization) or accesses the return address, one has the following
option: Let the sequence end with a NEXT instead of ;S and compile it
like a primitive (i.e., generate a pointer to the native code for the
sequence into the threaded code), not like a call to a colon
definition.

The result would be that the sequence performs only one NEXT and no
call overhead.

In regular code (where native code is generated), one could inline the sequence, avoiding even that one NEXT. We have planned this for the
time after the release of Gforth 1.0.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2026 CFP: http://www.euroforth.org/ef26/cfp.html
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:

Now if you have a sequence of primitives to generate in your
heap-allocated threaded code, such as "+ C@",

I'm not versed enough in Gforth's implementation to completely
understand this post, but is there currently a way to generate threaded
code in the heap?

I had thought threaded code in Gforth was mostly for debugging and to
make certain legacy hacks work, but that it is mostly ok to use the
native code compiler.

Yes I had confused MARKER with the older FORGET. I think they're pretty similar and both ugly.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> writes:

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:

Now if you have a sequence of primitives to generate in your
heap-allocated threaded code, such as "+ C@",

I'm not versed enough in Gforth's implementation to completely
understand this post, but is there currently a way to generate threaded
code in the heap?

The stuff that is allocated HERE is (a variant of) direct threaded
code; with dynamic native-code generation, these cells point to
dynamically generated native code, without they point into the code
segment of the binary.

I had thought threaded code in Gforth was mostly for debugging and to
make certain legacy hacks work, but that it is mostly ok to use the
native code compiler.

There is gforth-itc (indirect threaded code), and the stepping
debugger works only with that, and maybe there is some legacy software
that works with it and not with the other engines. There is no
dynamic native code generation here, because that does not work with indirect-threaded code (well, I can think of a way, but we have not
implemented it).

The gforth and gforth-native engine are basically direct-threaded and,
in the usual case, the threaded code points to dynamically generated
native code. Read all about it in:

@InProceedings{ertl&paysan25ef,
author = {M. Anton Ertl and Bernd Paysan},
title = {Code-Copying Compilation in Production---An Experience Report},
crossref = {euroforth25},
pages = {29--45},
url = {http://www.euroforth.org/ef25/papers/ertl.pdf},
url-slides = {http://www.euroforth.org/ef25/papers/ertl-slides.pdf},
OPTvideo = {},
note = {Paper also presented and published at KPS'25},
abstract = {A code-copying compiler implements a programming
language by concatenating code snippets produced by
a different compiler. This technique has been used
in Gforth since 2003, with code snippets generated
by GCC. We have solved various challenges: in
particular, which code snippets can be copied and
what to do about the others; and challenges posed by
changes in compilers. The performance of Gforth is
similar to that of SwiftForth, a commercial system
with a conventional compiler; the implementation
effort is comparable to 1--2 targets for SwiftForth.}
}

@Proceedings{euroforth25,
editor = {M. Anton Ertl},
title = {41st EuroForth Conference},
booktitle = {41st EuroForth Conference},
year = {2025},
key = {EuroForth'25},
url-proceedings = {http://www.euroforth.org/ef25/papers/proceedings.pdf}
}

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2026 CFP: http://www.euroforth.org/ef26/cfp.html
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> wrote:

Paul Rubin <no.email@nospam.invalid> writes:

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

I wonder if it suffices to temporarily set DP (the gforth dictionary
pointer, not documented in the gforth manual) to point into the
allocated memory. To allocate the correct sized block, maybe compile
twice. Something like (untested):

: compile-str { a u -- xt newloc }
HERE { old-here }
a u EVALUATE ( compile into dictionary )
DROP \ don't need the xt this time
ALIGN HERE old-here - { size }
size ALLOC { newloc }
newloc DP ! \ now HERE points to the heap
a u EVALUATE ( xt ) \ re-do the compilation
old-here DP ! \ restore dictionary
newloc ( xt newloc)
;

The xt and newloc both have to be kept around, so the xt can be executed
and so the GC can find the allocated block, but both can be copied into
some other ALLOC'd object which the GC will scan.

Alternatively, if the compiler works in a way that xt and newloc are
known to be the same address, there's no need to remember both.

This is how it works out in ciforth, using a temporary switch to an alternative area (also used for discarding the assembler, or build a prototype object.)
I use a super lightweight noname with the property that the first address
can serve as an xt. The normal ; is not usable. (It is a design error to use
'; that unsmudges the latest definition, that nowhere to be seen. )
ALLOCATE is under control of forth. That make SIZE possible. Also arbitrary buffers can be used for heap space, for example a mapped file.
INIT-ALLOC arbitrarily uses a quarter of the dictionary space for heap. Afterwards the definition is trimmed to size.
I hope you understand the REGRESS : S: announces the result of the test

WANT REGRESS ALLOCATE INIT-ALLOC SWAP-DP DO-DEBUG
: :NONAME HERE 'TASK @ , HERE CELL+ , ] ;
: ;; POSTPONE EXIT POSTPONE [ ; IMMEDIATE

REGRESS :NONAME 1 2 4 5 + . ;; EXECUTE S: 1 2

INIT-ALLOC
1,000,000 ALLOCATE THROW
FAR-DP ! \ Allocated area is now HERE.
SWAP-DP
:NONAME 1 2 4 5 + . ;; HERE OVER - RESIZE THROW
SWAP-DP
CONSTANT XT1
REGRESS XT1 EXECUTE S: 1 2
XT1 SIZE .
XT1 overhead - _free? . \ Internal function.
XT1 FREE THROW
XT1 overhead - _free? .

~/PROJECT/ciforths/ciforth: cat opop.frt | lina -a

S[ ] OK
S[ ] OK 9 ;; :NONAME 1 2 4 5 + . ;; EXECUTE S: 1 2 \ PASSED
S[ ] OK
S[ ] OK
S[ 4263296 ] OK
S[ ] OK
S[ ] OK
S[ 4263296 ] OK
S[ 4263296 ] OK
S[ ] OK 9 XT1 XT1 EXECUTE S: 1 2 \ PASSED

S[ ] OK 88
S[ ] OK 0
S[ ] OK
S[ ] OK -1

--
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

On 6/26/26 13:54, Paul Rubin wrote:

Let's say I convert a string into an anonymous function:

." :noname 3 3 + ;" EVALUATE

This leaves an xt on the stack. The compiled code lives in the
dictionary, right? So if I dynamically create a lot of such xt's for temporary use, that's a memory leak? If each one is supposed to be used
and released immediately after creation, FORGET could rewind the
dictionary to the right place, but maybe the required lifetimes are more complex.

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

This would be the garbage collected heap using Anton's GC, if it
matters.

Why to want this? Muhahaha.

kForth has a dynamic dictionary. All compilations are to heap memory.
This means there is no HERE address and the "," and "C," operators are
not supported -- there are simple workarounds. Both ALLOT and ALLOCATE
obtain memory from the heap, so it is possible to allot arbitrary size
blocks of memory without a size limit except that imposed by the OS.

For more info, see the User's Manual, 4.6 "Special Fearures", for kForth-64/-32/-Win32 at

https://ccreweb.org/software/kforth.html

--
Krishna Myneni

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

On 6/30/26 06:34, Krishna Myneni wrote:
..

For more info, see the User's Manual, 4.6 "Special Fearures", for kForth-64/-32/-Win32 at

https://ccreweb.org/software/kforth.html

Correction:

https://ccreweb.org/software/kforth/kforth.html

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

On 26 Jun 2026 at 20:54:38 CEST, "Paul Rubin" <no.email@nospam.invalid> wrote:

Let's say I convert a string into an anonymous function:

." :noname 3 3 + ;" EVALUATE

This leaves an xt on the stack. The compiled code lives in the
dictionary, right? So if I dynamically create a lot of such xt's for temporary use, that's a memory leak? If each one is supposed to be used
and released immediately after creation, FORGET could rewind the
dictionary to the right place, but maybe the required lifetimes are more complex.

So I'm wondering if there's a reasonable way to compile the code into
heap memory instead. It's ok if it's gforth specific I guess.

VFX Forth supports nested transient areas for code. See the file Examples/transients.fth. It only requires a fairly traditional use of HERE and the vafriable DP. The source code follows below.

Stephen

\ transients.fth - transient words

((
Copyright (c) 2025, 2026
Wodni & Pelc GmbH
Oedenstrasse 21/6/9
1210 Wien
Austria

Copyright (c) 2019
MicroProcessor Engineering
133 Hill Lane
Southampton SO15 5AF
England

tel: +44 7803 903612
www: vfxforth.com www.mpeforth.com


To do
=====

Change history
==============
))

only forth definitions
decimal

\ ===========
\ *! transientareas
\ *T Transient Areas
\ ===========

ErrDef bad_transientArea "Not in transient area"

64 kb value /TA \ -- len
\ Size of a transient area.

0 value anchorTR \ -- addr|0
\ *G Anchors the linked list of transient areas.

struct /transient \ -- len
\ *G Defines the head of the transient region.
int tr.link \ previous transient area in chain
int tr.prevDP \ previous DP
int tr.idata \ IDATA state
int tr.sp \ SP on entry
int tr.s0 \ S0 on entry
end-struct

: [TR \ --
\ *G Allocate a new transient area
sp@ >r \ stack pointer
anchorTR /TA protAlloc \ get space, THROW on error
tuck tr.link ! \ save previous transient area
dp @ over tr.prevDP ! \ save previous DP
idata? over tr.idata ! \ IDATA state
r> over tr.sp ! \ preserve old stack pointer
S0 @ over tr.s0 ! \ preserve old S0
dup -> anchorTR \ save new transient area
/transient + DP ! \ set new DP (HERE).
0 -> idata? \ no IDATA
0 0 0 0 0 0 0 0 \ guard cells on data stack
sp@ s0 ! \ reset S0
;

: TR] \ --
\ *G End the current transient area and remove it
anchorTR \ -- ta ; last TA
dup 0= bad_transientArea ?throw \ check transient area
dup /transient + here trim-dictionary \ forget transient words
dup tr.idata @ -> idata? \ restore IDATA state
dup tr.prevDP @ dp ! \ restore HERE
dup tr.sp @ >r \ previous SP
dup tr.s0 @ S0 !
dup tr.link @ swap protFree -> anchorTR \ unlink and free
r> sp! \ restore stack
;


\ ======
\ *> ###
\ ======

decimal
--
Stephen Pelc, stephen@vfxforth.com
Wodni & Pelc GmbH
Vienna, Austria
Tel: +44 (0)7803 903612, +34 649 662 974 http://www.vfxforth.com/downloads/VfxCommunity/
free VFX Forth downloads
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.forth

Paul Rubin <no.email@nospam.invalid> wrote:

understand this post, but is there currently a way to generate threaded
code in the heap?

Toad (FigForth) also use switch DP to compile in external space.
"ecomp" /demo

Demo External Compile

-ERRORS Setup external compile error handling
{ ... } Perform an external compile and store action in {}
{} Deferred word to execute external compiled action
{ERR} Print error message "{} is unset"
{FIN} Assign {ERR} to {}
E Execute {} once and perfrom {FIN}
i. Item mark that also prints "-->"


-ERRORS Setup error handling

-ERRORS \ enable external compile error handling
i. errs. -->
ABORT status:
WARNING -1
(ABORT) UABORT
UABORT (ERR_ECOMP)
ERRTRACE Disabled
ERRCUT Disabled


Error case: {} unset

{fin} \ clear external word
-ECUT \ enable cut on error
cut: i. {} \ a fault -->
(?) {} not set
{}? ABORTED
CURRENT and CONTEXT are WRK PREVOC is WRK BASE: 10
Latest: errs. HERE : 134596928 UNUSED : 11702
S?: 27
---
R: 134585300 CUTTING


E EXECUTE {} ONCE ONLY

{ ." HELLO WORLD " }
i. E --> HELLO WORLD


EXECUTE {} MULTIPLE TIMES

{ IF ." BILL " ELSE ." BOB " ENDIF }
i. 0 {} --> BOB
i. 1 {} --> BILL
{FIN}


REPLACE {}

{ ." BAR " }
i. {} --> BAR
{ ." BOO " }
i. {} --> BOO
{FIN}

-fin-
OK

--
me

--- Synchronet 3.22a-Linux NewsLink 1.2