In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <101f9oo$18edp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

substr() makes a copy. This is clear in the code.

It's almost impossible to do this via read-only descriptor.
Consider something like

x = substr($0, 10, 15)
getline
print x

Gawk manages the storage such that for something like
your example the copy will be released after index()
returns a value.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 31.05.2025 21:07, Mack The Knife wrote:

In article <101f9oo$18edp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

substr() makes a copy. This is clear in the code.

Okay. Thanks for checking that!

It's almost impossible to do this via read-only descriptor.
Consider something like

x = substr($0, 10, 15)
getline
print x

Well, it's possible to do that with a descriptor if GNU
Awk had a delayed/lazy evaluation principle implemented.
(Before 'getline' invalidates $0 a copy is necessary, of
course.)

(It's been reported that there's some optimizations in
GNU Awk implemented, so it could have also be the case
here. That's why I'm asking.)

Gawk manages the storage such that for something like
your example the copy will be released after index()
returns a value.

As said, I'm working on a huge string of data. What are
other options to efficiently work on substring parts of
the data? With the result of your code-check I don't see
a chance to achieve that with GNU or maybe any Awk using
only standard functionality.

Okay, maybe I could write an extension to work on memory
mapped files - the data originally stems from a file -
and seek/read through "C" mechanisms. (But that's huge
effort compared to some natively available function. And
then I'd probably better implement that straightly in "C"
instead of using Awk, in the first place, since I'd have
to implement the GNU Awk Extension anyway in "C".)

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2025-05-31, Mack The Knife <mack@the-knife.org> wrote:

In article <101f9oo$18edp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

substr() makes a copy. This is clear in the code.

It's almost impossible to do this via read-only descriptor.

It's entirely possible, with just reference counted memory management.
Under the hood, you have two kinds of string types: primary strings, and strings which are views displaced into other strings. The displacing
strings own a reference count on the target string, and indicate the
offset and range.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 31.05.2025 21:07, Mack The Knife wrote:

In article <101f9oo$18edp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

substr() makes a copy. This is clear in the code.

Okay. Thanks for checking that!

...

Okay, maybe I could write an extension to work on memory
mapped files - the data originally stems from a file -
and seek/read through "C" mechanisms. (But that's huge
effort compared to some natively available function. And
then I'd probably better implement that straightly in "C"
instead of using Awk, in the first place, since I'd have
to implement the GNU Awk Extension anyway in "C".)

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

--
Ben.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <87h60zrbea.fsf@bsb.me.uk>, Ben Bacarisse <ben@bsb.me.uk> wrote: ...

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

The thing about writing GAWK extensions is that the first one is hard,
because it is all new stuff to learn (and you have to establish your own conventions for how your extensions are going to look, code-wise). But once you've written one, the second is much easier and it gets progressively
easier after that. You want to get to a point where to write a new
extension, you just take the previous one you wrote, copy it to a new name,
do a global search/replace of "oldname" with "newname", delete the
operative code from the old extension, replace it with the new operative
code (which is usually a very small amount of actual code) - and, you're done.

I think that an index()-like function that starts the search at a given
offset into the main string (say, position 900000 of a one million
character string) would be both a good idea and pretty easy to implement as
an extension. I could do it - at least modulo i18n. My version would be
C locale only. Note that index() doesn't do regexps; the search would be straight string only - using strstr() (or strcasestr() if IGNORECASE is
set).

By the way, if you find the substring at position 900005 (i.e., the 5th (*) char of the searched string), should the function return 5 or 900005?

(*) Or 6th; I'm not sure of my exact notation at this point.

--
People often ask what is the difference between liberals and conservatives.
It is this. Libs see the government helping them and are OK with the government also helping other people. Cons see the government screwing them and are OK with that as long as the government is also screwing other people.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 01.06.2025 12:42, Ben Bacarisse wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 31.05.2025 21:07, Mack The Knife wrote:

In article <101f9oo$18edp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

In the context p=index(substr(t,s),r)
it would not be necessary to copy the substr(t,s),
the index() function could operate on the original
using some access "descriptor" (say, a pointer and
a length) in read-only mode.

Will (GNU) Awk do a copy of the data value or does
it use a read-only descriptor access to the already
existing substring of variable "t"?

Currently I'm playing with some huge data and copies
of MB sized data is costly (if it's repeatedly done
with various substr() subscripts).

substr() makes a copy. This is clear in the code.

Okay. Thanks for checking that!

...

Okay, maybe I could write an extension to work on memory
mapped files - the data originally stems from a file -
and seek/read through "C" mechanisms. (But that's huge
effort compared to some natively available function. And
then I'd probably better implement that straightly in "C"
instead of using Awk, in the first place, since I'd have
to implement the GNU Awk Extension anyway in "C".)

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

This, first of all, sounds like a good idea! It would make it
unnecessary to (mis-)use the substr() function as (sort of) a
costly copying-descriptor.[*]

I'm unsure about using an extension here. Would there be a name
clash between an built-in index(haystack,needle) function and an
extension index(haystack,needle,start) function? Should they be
separate functions in the first place? (I don't think so.)

In the past new extended functionality was supported by additional
optional parameters in the core Awk code. (Which seems to be the
best place [for optional controlling arguments].) - There's quite
some examples where it seems to have worked well with optional
parameters in the core functions. The changes were obviously local
and the frightening side-effects were not arising, it seems.

But we've read the recent links (with the interview) or already
know what Arnold thinks about that; and it is ambivalent. For one
there was a complaint about quality issues of contributed code
and the maintainer's reluctance to add such code - which is very understandable! But then there's also the problem that maintainers
don't want to "jump" when arbitrary wishes on functionality arise.

Janis

[*] N.B.: To be consistent it should probably support a substring,
as in index(haystack,needle [,start[,end]]), since the application
example given above p=index(substr(t,s),r) in its generalized form
would have been p=index(substr(t,s,e),r).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <101hecq$22ab2$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
...

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

This, first of all, sounds like a good idea! It would make it
unnecessary to (mis-)use the substr() function as (sort of) a
costly copying-descriptor.[*]

I'm unsure about using an extension here. Would there be a name
clash between an built-in index(haystack,needle) function and an
extension index(haystack,needle,start) function? Should they be
separate functions in the first place? (I don't think so.)

Nobody is talking about changing the index() built-in function.

This would be a brand new function, written as an extension library.

You could name it something like index_ex() if you like, or you could give
it a brand new name (**).

Although I have not done this with the extension functions I've already
written (and have widely deployed, so obviously, as the saying goes, it's
way too late to change it), if I had it to do over again, I'd probably have enforced a convention that my functions always start with some identifying prefix - to make it clear that these were written by me, not built-ins. I
do that when writing in other scripting languages; I should have done so in AWK. Perhaps, the new namespaces feature would address this (*).

(*) I have not yet taken the time to fully acclimate to the concepts and
start using namespaces in my own code. I'm still working on getting the
kinks and quirks out of PMA...

(**) My conception of how this would be implemented would handle the "start"-only case (just add the offset to the "haystack" arg of strstr() -
with error checking to make sure it doesn't overflow, of course).
Implementing "end" would be a bit trickier (but not much).

--
Liberals live in a fantasy world where (street) criminals are good people.

Conservatives live in a fantasy world where businessmen are good people.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 01.06.2025 14:06, Kenny McCormack wrote:

In article <101hecq$22ab2$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
...

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

This, first of all, sounds like a good idea! It would make it
unnecessary to (mis-)use the substr() function as (sort of) a
costly copying-descriptor.[*]

I'm unsure about using an extension here. Would there be a name
clash between an built-in index(haystack,needle) function and an
extension index(haystack,needle,start) function? Should they be
separate functions in the first place? (I don't think so.)

Nobody is talking about changing the index() built-in function.

Well, I was.

asort(source [, dest [, how ] ])
gensub(regexp, replacement, how [, target])
gsub(regexp, replacement [, target])
match(string, regexp [, array])
patsplit(string, array [, fieldpat [, seps ] ])
split(string, array [, fieldsep [, seps ] ])
substr(string, start [, length ]) # already standard Awk
close(filename [, how])
mktime(datespec [, utc-flag ])
strftime([format [, timestamp [, utc-flag] ] ])

...there are already standard Awk functions (sensibly!) extended by
optional arguments, and also GNU Awk specific functions that either
have from the beginning optional arguments or got extended later.

This would be a brand new function, written as an extension library.

You could name it something like index_ex() if you like, or you could give
it a brand new name (**).

None of that I'd find a good solution given the existence of index().
(YMMV.)

[...]

(**) My conception of how this would be implemented would handle the "start"-only case (just add the offset to the "haystack" arg of strstr() - with error checking to make sure it doesn't overflow, of course). Implementing "end" would be a bit trickier (but not much).

Yes, I'd second that.

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 01.06.2025 13:53, Kenny McCormack wrote:

In article <87h60zrbea.fsf@bsb.me.uk>, Ben Bacarisse <ben@bsb.me.uk> wrote: ...

An alternative (depending on the context) would be to consider an
extension that provides an index function with a third argument giving
the initial offset. I've not looked at how extensions get access to
GAWK strings, so this many not be as easy as it sounds, but I would
guess that it might be relatively simple to do.

The thing about writing GAWK extensions is that the first one is hard, because it is all new stuff to learn (and you have to establish your own conventions for how your extensions are going to look, code-wise). [...]

You are describing the individual practical accustoming to writing
own extensions.

Okay. My viewpoint was another.

It's IMO a problem if folks write own index() extension in his/her
own version and code quality. We see proliferations of own versions
in all areas of IT; and I don't see it as a desirable goal.

I wouldn't want two (or three) 'match' functions (match(), match_a(), match_ae()) or similar.[*]

If a function provides actually the same basic task, and it should
get controlled (like program options[**]) with optional arguments,
that would be what I'd think be the right way.

By the way, if you find the substring at position 900005 (i.e., the 5th (*) char of the searched string), should the function return 5 or 900005?

(*) Or 6th; I'm not sure of my exact notation at this point.

I don't think the decision is crucial since (I think) you could
derive one value from the other (provided the given arguments).

The behavior should probably be discussed to find the advantages
of one or the other option. Yet I'm not sure whether the usefulness
of such a match() function extension is commonly accepted.


(Anyway. Core evolutions are deprecated. And it won't happen.)

Janis

[*] I find it already a bit, umm, strange to have three different
substitution functions in GNU Awk (two historic/standard and one
somewhat generalized and extended).

[**] Do we need grep, egrep, fgrep - on my system they are not
even hardlinks -, or should grep be used with options grep -E,
grep -F ?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <101hlls$24vmc$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
...

You are describing the individual practical accustoming to writing
own extensions.

Well, yeah. I have already scaled the hurdles to which I refer.
I was trying to give you a pointer towards getting there yourself.
You are, of course, free to ignore and/or mock my attempts.

Okay. My viewpoint was another.

It's IMO a problem if folks write own index() extension in his/her
own version and code quality. We see proliferations of own versions
in all areas of IT; and I don't see it as a desirable goal.

This is the fundamental issue in computing. Is it better for people to
be free to do their own versions of things, or are we obliged to coalesce around versions written by "professionals"? It all goes back to that
famous "letter to the editor", written by one William Gates III, arguing strongly for the later.

The general consensus in the FOSS world is towards the former, but from
time to time, I see certain people here arguing for the later.

(Anyway. Core evolutions are deprecated. And it won't happen.)

Exactly. That's the bottom line. That's why both Ben and I are discussing this in the context of a user-written extension function/library.

You are, of course, free to continue whining that it should be changed in
the core, by the official developers. Lots of luck with that.

[*] I find it already a bit, umm, strange to have three different >substitution functions in GNU Awk (two historic/standard and one
somewhat generalized and extended).

That's the way it is.
It can't change (either in the general or the specific).

[**] Do we need grep, egrep, fgrep - on my system they are not
even hardlinks -, or should grep be used with options grep -E,
grep -F ?

Personally, I don't think "grep" should ever be used at all any more -
except for certain specific, controlled, simple cases. Much the same
argument can be (and has been, on this very newsgroup) applied to the use
of "sed".

Unix has a lot of historical cruft commands that aren't really relevant
any more - e.g., join, comm, sed, sort (yes, I'm pushing the envelope a bit with the last two) - that really shouldn't be used anymore, since their functionality has been subsumed by more complete scripting languages (AWK,
of course, but also perl, etc if you are so inclined).

--
Pensacola - the thinking man's drink.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <101fv4s$1g5c8$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Okay, maybe I could write an extension to work on memory
mapped files - the data originally stems from a file -
and seek/read through "C" mechanisms. (But that's huge
effort compared to some natively available function. And
then I'd probably better implement that straightly in "C"
instead of using Awk, in the first place, since I'd have
to implement the GNU Awk Extension anyway in "C".)

Janis

You could check w/Arnold about his consulting rates. Perhaps
he would do a patch for you, or write an extension function
for you.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <683e9c84$0$691$14726298@news.sunsite.dk>,
Mack The Knife <mack@the-knife.org> wrote:

In article <101fv4s$1g5c8$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Okay, maybe I could write an extension to work on memory
mapped files - the data originally stems from a file -
and seek/read through "C" mechanisms. (But that's huge
effort compared to some natively available function. And
then I'd probably better implement that straightly in "C"
instead of using Awk, in the first place, since I'd have
to implement the GNU Awk Extension anyway in "C".)

Janis

You could check w/Arnold about his consulting rates. Perhaps
he would do a patch for you, or write an extension function
for you.

Yeah, I think that aspect of it has already been covered.

The thing is - you have to understand that the mentality of a certain kind
of poster is to not be oriented towards: "How can I get my specific,
personal problem solved?"

But rather, it is: "Can I get (*) my personal pet feature into the GAWK core?"

(*) And by "get", I mean "nag the core developers into adding it".

And don't get me wrong, this approach is not without merit. Getting it
into the core benefits everybody, not just the poster (nagger). Suppose
that GAWK had a misfeature where when you added 2 and 2, you get 3.79. If
we could nag them into changing it so that adding 2 and 2 produced 4
instead, that would benefit everybody and it would be worthwhile for us all
to do that nagging (rather than have one of us here on the newsgroup cook
up a personalized solution that benefits only the poster).

But, getting back to topic, it (the "mentality" to which I refer above)
does represent a stark difference of opinion and purpose to what most of us here on this and other newsgroups hold to.

Anyway, there is more I could say on this subject, but I think I've said
enough for now. But it is likely that an extension to do exactly this will
end up getting written by me and posted here soon - given that I've already spec'd it out pretty thoroughly.

--
[Donald] Trump didn't have it all handed to him by his parents,
like Hillary Clinton did.

- Some dumb cluck in Ohio; featured in Michael Moore's "Trumpland" -

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 01.06.2025 16:17, Kenny McCormack wrote:

[...]

You are, of course, free to continue whining that it should be changed in
the core, by the official developers. Lots of luck with that.

I don't know where you've got that from. - I was merely suggesting
an IMO sensible and compatible change of a function in a way other
functions (standard and non-standard) have already been extended.
I think it belongs in the core and I was suggesting and discussing
that.

If the relevant official developers don't think so then it is as it
is and they may just ignore my suggestion. I have no stakes in GNU
Awk, only (maybe academical) interest, so I even don't really care
much beyond that.

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 03.06.2025 08:56, Mack The Knife wrote:

You could check w/Arnold about his consulting rates. Perhaps
he would do a patch for you, or write an extension function
for you.

This is an extremely stupid comment in so many ways!

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <1022d33$3copv$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 03.06.2025 08:56, Mack The Knife wrote:

You could check w/Arnold about his consulting rates. Perhaps
he would do a patch for you, or write an extension function
for you.

This is an extremely stupid comment in so many ways!

Just out of curiousity, why?

If you don't wish to write code yourself, but you want to have
access to a feature, what's wrong with hiring someone to do the
work for you?

A patch to the source code that you could enable to build
a version of gawk with your feature would work well.

Or an extension function might do the trick.

If you really don't need the feature after all, then what
was the point in starting the discussion in the first place?

I am not being sarcastic, I am genuinely curious.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 08.06.2025 14:35, Mack The Knife wrote:

In article <1022d33$3copv$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 03.06.2025 08:56, Mack The Knife wrote:

You could check w/Arnold about his consulting rates. Perhaps
he would do a patch for you, or write an extension function
for you.

This is an extremely stupid comment in so many ways!

Just out of curiousity, why?

I'm not interested in meta-chat. - But you can derive part
of the answer already from my recent reply to Kenny's post
(only a few days ago). HTH.

Janis

[...]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <102bh1b$1sskv$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
...

I'm not interested in meta-chat. - But you can derive part
of the answer already from my recent reply to Kenny's post
(only a few days ago). HTH.

Yes. I agree. I have included the text below - for Mack's benefit.
It summarizes quite accurately your position on this issue and should be
taken as the final word.

--- Cut Here ---
I don't know where you've got that from. - I was merely suggesting
an IMO sensible and compatible change of a function in a way other
functions (standard and non-standard) have already been extended.
I think it belongs in the core and I was suggesting and discussing
that.

If the relevant official developers don't think so then it is as it
is and they may just ignore my suggestion. I have no stakes in GNU
Awk, only (maybe academical) interest, so I even don't really care
much beyond that.

Janis
--- Cut Here ---

Mack's confusion arises from being confused about what your motives are.
(I could - and probably will at some later time - expand upon what I mean
by this, but for now, the previous sentence will have to suffice)

--
In American politics, there are two things you just don't f*ck with:

1) Goldman Sachs
2) The military/industrial complex

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)