On Tue, 24 Sep 2024 12:20:42 +0000, albert@spenarnc.xs4all.nl wrote:
[..]

The following result up to 8 Gbyte are more sensible,
up to 8 Gbyte the second time is faster.

root@cherry:/home/albert/PROJECT/ciforths/ciforth# nice -20 lina16G++
WANT ELAPSED
....
OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
537.777mS OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
253.468mS OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
257.628mS OK
MARK-TIME HERE 2,000,000,000 ERASE ELAPSED .mS
781.149mS OK
MARK-TIME HERE 2,000,000,000 ERASE ELAPSED .mS
503.593mS OK
MARK-TIME HERE 4,000,000,000 ERASE ELAPSED .mS
1916.234mS OK
MARK-TIME HERE 4,000,000,000 ERASE ELAPSED .mS
1554.072mS OK
MARK-TIME HERE 8,000,000,000 ERASE ELAPSED .mS
8059.426mS OK
MARK-TIME HERE 8,000,000,000 ERASE ELAPSED .mS
5519.860mS OK

This is exactly the problem that I encountered on
Windows: allocating GBytes of memory is a lot faster
the second time you do it, which points to an OS
memory-management feature. For me it happened when
the allocated memory size was reaching 1GB on a 32GB
system with 20GB still free, so your Linux numbers are
actually quite good.

I hoped to improve matters with the Working-Set
size (which is explicitly shown on Windows 11's
Task manager) but was not successful at the time
(will look again).

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <f9e2a709183796c9cee0d60d24b0cf89@www.novabbs.com>,
mhx <mhx@iae.nl> wrote:

On Tue, 24 Sep 2024 12:20:42 +0000, albert@spenarnc.xs4all.nl wrote:
[..]

The following result up to 8 Gbyte are more sensible,
up to 8 Gbyte the second time is faster.

root@cherry:/home/albert/PROJECT/ciforths/ciforth# nice -20 lina16G++
WANT ELAPSED
....
OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
537.777mS OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
253.468mS OK
MARK-TIME HERE 1,000,000,000 ERASE ELAPSED .mS
257.628mS OK
MARK-TIME HERE 2,000,000,000 ERASE ELAPSED .mS
781.149mS OK
MARK-TIME HERE 2,000,000,000 ERASE ELAPSED .mS
503.593mS OK
MARK-TIME HERE 4,000,000,000 ERASE ELAPSED .mS
1916.234mS OK
MARK-TIME HERE 4,000,000,000 ERASE ELAPSED .mS
1554.072mS OK
MARK-TIME HERE 8,000,000,000 ERASE ELAPSED .mS
8059.426mS OK
MARK-TIME HERE 8,000,000,000 ERASE ELAPSED .mS
5519.860mS OK

This is exactly the problem that I encountered on
Windows: allocating GBytes of memory is a lot faster
the second time you do it, which points to an OS
memory-management feature. For me it happened when
the allocated memory size was reaching 1GB on a 32GB
system with 20GB still free, so your Linux numbers are
actually quite good.

I hoped to improve matters with the Working-Set
size (which is explicitly shown on Windows 11's
Task manager) but was not successful at the time
(will look again).

Hebish explained it for a great deal.
Clearing a 16 Gbyte area was done with with a large
fraction of memory somehow used.
free reveals a 10 Gbyte memory available.
That nicely explains the discrepancy between 8 Gbyte
and 16 Gbyte clearance.

-marcel

--
Temu exploits Christians: (Disclaimer, only 10 apostles)
Last Supper Acrylic Suncatcher - 15Cm Round Stained Glass- Style Wall
Art For Home, Office And Garden Decor - Perfect For Windows, Bars,
And Gifts For Friends Family And Colleagues.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 23 Sep 2024 12:18:42 +0000, albert@spenarnc.xs4all.nl wrote:

[..]

8 Gigabyte would take 2 seconds. Although the whole program fits
comfortably in RAM, it takes 30 times longer.
What gives?

I don't know (yet) how it works in Linux, but in Windows there
is something called 'Working-Set'. If I exceed that value,
things slow down.

It is quite difficult (if not impossible) to increase the
Working-Set without extensive knowledge (believe me, I tried).
In iForth I have about 0.7GB now, the WSL has 2GB, FireFox 1.6GB.
This is on a 32GB system with 20GB free.

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 23 Sep 2024 17:51:58 +0000, mhx wrote:

I don't know (yet) how it works in Linux, but in Windows there
is something called 'Working-Set'. If I exceed that value,
things slow down.

It is quite difficult (if not impossible) to increase the
Working-Set without extensive knowledge (believe me, I tried).

Have you tried the SetProcessWorkingSetSize function
(memoryapi.h)?

It sets the minimum and maximum working set sizes for the
specified process, they say...

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 23 Sep 2024 18:19:44 +0000, minforth wrote:

On Mon, 23 Sep 2024 17:51:58 +0000, mhx wrote:

I don't know (yet) how it works in Linux, but in Windows there
is something called 'Working-Set'. If I exceed that value,
things slow down.

It is quite difficult (if not impossible) to increase the
Working-Set without extensive knowledge (believe me, I tried).

Have you tried the SetProcessWorkingSetSize function
(memoryapi.h)?

It sets the minimum and maximum working set sizes for the
specified process, they say...

Yes, but...
"The handle must have the PROCESS_SET_QUOTA access right.
For more information, see Process Security and Access Rights"

And there the rabbit hole opens...

It does appears to be better documented than the last time
I looked:
"PROCESS_SET_QUOTA (0x0100) Required to set memory limits using SetProcessWorkingSetSize."
... should be used with
"To retrieve a process's security descriptor, call the
GetSecurityInfo function. To change a process's security
descriptor, call the SetSecurityInfo function."

I expect it wants the executable run by an Administrator,
but that would only be a temporary nuisance.

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 23 Sep 2024 19:10:08 +0000, mhx wrote:

On Mon, 23 Sep 2024 18:19:44 +0000, minforth wrote:

Have you tried the SetProcessWorkingSetSize function
(memoryapi.h)?

It sets the minimum and maximum working set sizes for the
specified process, they say...

Yes, but...

I was of the perhaps naive opinion that it would be sufficient
to get the actual min/max quota using

GetProcessWorkingSetSize(..)

then increase the max quotum and call

SetProcessWorkingSetSize(..)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 30 Sep 2024 9:53:32 +0000, albert@spenarnc.xs4all.nl wrote:

I played around with it some more on Windows 11.

FORTH> .TICKER-INFO
AMD Ryzen 7 5800X 8-Core Processor, timer & PROCESSOR-CLOCK of 4192MHz
Peak WorkingSetSize = 650 kB
Current WorkingSetSize = 631 kB
Total memory size = 31 GB
Available memory size = 16 GB
ok
FORTH> 0 VALUE am1 ok
FORTH> TIMER-RESET #2000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.215 seconds elapsed. ok

FORTH> TIMER-RESET #2000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.155 seconds elapsed. ok

Faster the second time.

FORTH> TIMER-RESET #8000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.902 seconds elapsed. ok

FORTH> TIMER-RESET #8000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.616 seconds elapsed. ok

FORTH> TIMER-RESET #8000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.583 seconds elapsed. ok

FORTH> TIMER-RESET #8000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.586 seconds elapsed. ok

With 8 GB the OS apparently is busy in the background and takes 4x as
long, a linear increase.

FORTH> TIMER-RESET #1000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.076 seconds elapsed. ok

If we go back to 1 GB the memory/diskspace is apparently pooled.

FORTH> TIMER-RESET #12000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 1.120 seconds elapsed. ok

The timing suggests that the pool is increased, not build from 0.

FORTH> TIMER-RESET #12000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.908 seconds elapsed. ok
FORTH> TIMER-RESET #12000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.881 seconds elapsed. ok

FORTH> TIMER-RESET #12000000000 ALLOCATE ?ALLOCATE TO am1 .ELAPSED
am1 FREE ?ALLOCATE 0.883 seconds elapsed. ok

FORTH> .TICKER-INFO
AMD Ryzen 7 5800X 8-Core Processor, timer & PROCESSOR-CLOCK of 4192MHz
Peak WorkingSetSize = 12075 kB
Current WorkingSetSize = 631 kB
Total memory size = 31 GB
Available memory size = 16 GB

The WorkingSetSize reflects the program's size + allocated memory.
AFAIR, this was a lot worse a few years ago.
Mysteriously, available memory size is still 16 GB.

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 30 Sep 2024 15:21:57 +0000, mhx wrote:

On Mon, 30 Sep 2024 9:53:32 +0000, albert@spenarnc.xs4all.nl wrote:

[..]

FORTH> .TICKER-INFO
AMD Ryzen 7 5800X 8-Core Processor, timer & PROCESSOR-CLOCK of 4192MHz
Peak WorkingSetSize = 12075 kB
Current WorkingSetSize = 631 kB
Total memory size = 31 GB
Available memory size = 16 GB

The WorkingSetSize reflects the program's size + allocated memory.
AFAIR, this was a lot worse a few years ago.
Mysteriously, available memory size is still 16 GB.

Arrgh, this is not correct. 12075 kB is 12 MB, not 12 GB. The
workingsetsize increases, but the relation to allocated memory
is not obvious, nor is it clear that a larger Peak WorkingSetSize
means decreased allocation delays.

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Bug found - the units were wrong.

iSPICE> 13 tt ( allocate 13 GB )
1.000 seconds elapsed. ok
iSPICE> 13 tt .TICKER-INFO
1.113 seconds elapsed.
AMD Ryzen 7 5800X 8-Core Processor, timer & PROCESSOR-CLOCK of 4192MHz
Peak WorkingSetSize = 13982 MB
Current WorkingSetSize = 67 MB
Total memory size = 31 GB
Available memory size = 17 GB
Total paged memory size = 40 GB
Available paged memory size = 16 GB

I now use ...
SetProcessWorkingSetSizeEx(
hProcess,
1000000000, /* dwMinimumWorkingSetSize 1GB */
16000000000, /* dwMaximumWorkingSetSize 16GB */
QUOTA_LIMITS_HARDWS_MAX_DISABLE /* Flags */
);

This minimizes the time for a/the first (large)
allocation and discourages the OS to swap out
deallocated memory.

-marcel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)