• [ksh93u+m] Interactive subshell expansion disfunctional

    From Janis Papanagnou@21:1/5 to All on Tue May 21 18:57:47 2024
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    The latest observation is that in interactive mode with 'set -o vi'
    I was able to write a command substitution, say,

    ls $(date)

    and with the cursor positioned (e.g.) at the closing parenthesis I
    could expand the argument by Esc-* to obtain

    ls 2024\-05\-21

    This does not seem to work in ksh93u+m. - Is there a chance to get
    original ksh behavior back?

    Thanks.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Martijn Dekker@21:1/5 to All on Sat Jun 22 00:14:27 2024
    Op 21-05-2024 om 17:57 schreef Janis Papanagnou:
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    I would of course be very interested in learning what deficiencies (other than the below) you've been observing.

    The latest observation is that in interactive mode with 'set -o vi'
    I was able to write a command substitution, say,

    ls $(date)

    and with the cursor positioned (e.g.) at the closing parenthesis I
    could expand the argument by Esc-* to obtain

    ls 2024\-05\-21

    This does not seem to work in ksh93u+m. - Is there a chance to get
    original ksh behavior back?

    "Subshell expansion" was never meant to be a thing and is not documented anywhere. The behaviour was a bug; this was not supposed to happen. It also behaved inconsistently and crashed the shell in some cases.

    For further details and rationale, please see: https://github.com/ksh93/ksh/commit/7a2d3564 https://github.com/ksh93/ksh/issues/268

    The documented behaviour of ESC-* in emacs is:

    M-* Attempt pathname expansion on the current word. An asterisk is
    appended if the word doesn't match any file or contain any
    special pattern characters.

    Pathname expansion simply has no business executing code; that's not what it's for.

    I can see how the behaviour as you describe it could come in handy, though. We could treat it as a feature request for version 93u+m/1.1 and up. It would
    have to be implemented properly. If implemented it should probably also be bound to a new key combination.

    --
    || modernish -- harness the shell
    || https://github.com/modernish/modernish
    ||
    || KornShell lives!
    || https://github.com/ksh93/ksh

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Martijn Dekker on Sat Jun 22 04:10:05 2024
    Hi Martijn

    On 22.06.2024 01:14, Martijn Dekker wrote:
    Op 21-05-2024 om 17:57 schreef Janis Papanagnou:
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    I would of course be very interested in learning what deficiencies
    (other than the below) you've been observing.

    Unfortunately I haven't kept track. It's just that I noticed that
    a couple things that worked before do not any more. Partly because
    they were probably anyway only considered experimental, partly it
    were features that I'd suspect are not commonly used.

    Off the top of my head one was the support of ksh's alarm timers,
    and another thing was in context of discipline functions used on
    shell variables. (In the latter case I have no certainty whether
    it's a shell issue or whether I should have done this differently.
    I detected it with my "kosta" code, a ksh variant of Kaz' "basta"
    for bash, only with features from ksh implemented.) Another thing
    is how variables are expanded in Vi-mode.

    (I will need some time to reproduce and collect the information.)


    The latest observation is that in interactive mode with 'set -o vi'
    I was able to write a command substitution, say,

    ls $(date)

    (This should of course have been ls $(date +%F) for the output
    format shown below.)


    and with the cursor positioned (e.g.) at the closing parenthesis I
    could expand the argument by Esc-* to obtain

    ls 2024\-05\-21

    This does not seem to work in ksh93u+m. - Is there a chance to get
    original ksh behavior back?

    "Subshell expansion" was never meant to be a thing and is not documented anywhere.

    Kornshell had always things implemented (for experimentation or
    even for production) and not always documented it [in time or
    generally].

    The behaviour was a bug; this was not supposed to happen.

    (I cannot see where you derive that from.)

    All I can say is that because the docs had partly always been
    behind of time or inconsistent it's not necessarily a bug; there
    are more things implemented than documented. (Not sure about the
    "latest" 93u+ but that was an observation since my ksh88 days.)

    So if you want to say "all that isn't documented isn't supported"
    I can hardly argue with you and will (of course) accept that.

    It
    also behaved inconsistently and crashed the shell in some cases.

    The features I used behaved well. But I agree that some usages
    showed inconsistencies (a general issue with old ksh; that's
    why I'm so happy that you provided a thoroughly fixed ksh93u+m).


    For further details and rationale, please see: https://github.com/ksh93/ksh/commit/7a2d3564 https://github.com/ksh93/ksh/issues/268

    The documented behaviour of ESC-* in emacs is:

    I cannot speak for Emacs-mode since I'm doing everything in
    Vi-mode (as indicated above).


    M-* Attempt pathname expansion on the current word. An
    asterisk is
    appended if the word doesn't match any file or contain any
    special pattern characters.

    Pathname expansion simply has no business executing code; that's not
    what it's for.

    The expansion I mentioned was not the only expansion issue.

    But first let me explain a short difference between pathname
    expansion and command expansion as it was. If I now type

    x=$(date +%F)<Esc>*

    then <Esc>* is expanding all files. I would have expected (as
    it had been before with ksh93u+) that a filename expansion
    would appear if I had typed

    x=$(date +%F)*<Esc>*

    i.e. if I had provided a glob pattern that the Vi command
    <Esc>* could then expand, or if the cursor would be on a
    word (a file or command) to become expanded (by globbing
    and PATH expansion, respectively). Now in ksh93u+m both
    depicted commands expand the same way (and lists all files
    in the working directory; not very useful, and luckily
    there is an undo command to let that filename pollution
    vanish).

    (I think it's also okay, as it currently works already,
    to expand all filenames on a space x=$(date +%F) <Esc>*
    but I think [from my perspective] it's a rarer case.)

    Moreover, if the cursor is on the opening parenthesis the
    <Esc>* will insert an escaped '\*' despite being in command
    mode (and not in input mode).

    Variable expansion does also misbehave (I think already in
    original ksh93u+). With

    y='ABC DEF'

    z=$y<Esc>*

    will expand to

    z=ABC DEF

    and trying to add quotes

    z="$y"<Esc>*

    will do nothing. And going back one character to the 'y'
    and then typing <Esc>* will produce

    z=ABC DEF"

    In my daily work with the Kornshell these expansions were
    really useful (modulo the bugs/inconsistencies of course).
    But of course I cannot expect that features that are (maybe,
    haven't checked) not (not well) documented to be supported.
    Especially if there were (as you also say) inconsistencies
    or bugs already in the original Kornshell.


    I can see how the behaviour as you describe it could come in handy,
    though. We could treat it as a feature request for version 93u+m/1.1 and
    up. It would have to be implemented properly. If implemented it should probably also be bound to a new key combination.

    I found it intuitive as it could be used before in ksh93u+,
    so I'm not very keen if another key combination will be
    invented. (My muscle memory will certainly fool me then.)

    Wouldn't it be consistently describable depending on the
    context? (I think it would.) On a command fragment or in
    a command position a pathname expansion will be done, on
    other word fragments (or globbing characters) a globbing
    expansion will be done, on a $ prefixed variable name a
    variable expansion will be done (ideally with a correct
    quotung), and on a shell construct character (like the
    parentheses or dollar in $(...) a command expansion will
    be done. - This is not yet thought through, but it
    resembles the previous behavior modulo some adjustments
    to address some previously existing inconsistency.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Janis Papanagnou on Sat Jun 22 14:47:31 2024
    On 22.06.2024 14:15, Janis Papanagnou wrote:
    On 22.06.2024 04:10, Janis Papanagnou wrote:
    On 22.06.2024 01:14, Martijn Dekker wrote:
    Op 21-05-2024 om 17:57 schreef Janis Papanagnou:
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    I would of course be very interested in learning what deficiencies
    (other than the below) you've been observing.

    [...] another thing was in context of discipline functions used on
    shell variables. [...]

    [...]

    and the "kosta" code contains

    function LINES.set { .kosta.rows=${.sh.value} ; .kosta.update ;}
    function COLUMNS.set { .kosta.cols=${.sh.value} ;}
    function PS1.get { .kosta.compose ;}
    namespace kosta
    {
    ...
    }

    and it's activated by sourcing it from ~/.kshrc

    I can't tell whether I have missed something with discipline functs
    on built-in variables, how I used the namespaces, or something like
    that. (My tries to fix that in my program logic all failed.) Though
    the above "no parent" error messages did not show up with ksh93u+.
    And it can now be observed in various _script interaction_ contexts.

    There's another effect in this context.

    Previously with ksh93u+ the "kosta" code called '.kosta.compose' if
    'PS1.get' gets called; the compose function reconstructs the status
    line. If an error occurred or a signal arrived that was indicated
    in the status line as, say, "ERR 2!" or "SIG INT(2)!". After I hit
    the <Enter> key (an empty line) that part of the status message got
    cleared. - Not any more with ksh93u+m. It seem that the 'compose'
    function won't be called any more on an empty shell input line.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Janis Papanagnou on Sat Jun 22 14:15:19 2024
    On 22.06.2024 04:10, Janis Papanagnou wrote:
    On 22.06.2024 01:14, Martijn Dekker wrote:
    Op 21-05-2024 om 17:57 schreef Janis Papanagnou:
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    I would of course be very interested in learning what deficiencies
    (other than the below) you've been observing.

    Unfortunately I haven't kept track. It's just that I noticed that
    a couple things that worked before do not any more. Partly because
    they were probably anyway only considered experimental, partly it
    were features that I'd suspect are not commonly used.

    Off the top of my head one was the support of ksh's alarm timers,
    and another thing was in context of discipline functions used on
    shell variables. (In the latter case I have no certainty whether
    it's a shell issue or whether I should have done this differently.

    An example for the above mentioned effect shows up like that...

    $ pushftp
    pushftp: LINES.set: line 44: .kosta.rows=24: no parent
    pushftp: COLUMNS.set: line 45: .kosta.cols=79: no parent
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]

    where pushftp is an 'expect' wrapper script

    file=${1:?"${usage}"}
    path=${2:?"${usage%/}"}
    domain=${3:-"some-domain"}
    path=${domain}/${path}
    expect <<EOT
    ...
    EOT

    and the "kosta" code contains

    function LINES.set { .kosta.rows=${.sh.value} ; .kosta.update ;}
    function COLUMNS.set { .kosta.cols=${.sh.value} ;}
    function PS1.get { .kosta.compose ;}
    namespace kosta
    {
    ...
    }

    and it's activated by sourcing it from ~/.kshrc

    I can't tell whether I have missed something with discipline functs
    on built-in variables, how I used the namespaces, or something like
    that. (My tries to fix that in my program logic all failed.) Though
    the above "no parent" error messages did not show up with ksh93u+.
    And it can now be observed in various _script interaction_ contexts.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Martijn Dekker@21:1/5 to All on Tue Jul 2 04:35:28 2024
    Op 22-06-2024 om 13:15 schreef Janis Papanagnou:
    On 22.06.2024 04:10, Janis Papanagnou wrote:
    On 22.06.2024 01:14, Martijn Dekker wrote:
    Op 21-05-2024 om 17:57 schreef Janis Papanagnou:
    I recently replaced all original ksh93u+ by ksh93u+m, and meanwhile
    noticed quite some deficiencies, things that worked in original ksh
    but not any more in ksh93u+m.

    I would of course be very interested in learning what deficiencies
    (other than the below) you've been observing.

    Unfortunately I haven't kept track. It's just that I noticed that
    a couple things that worked before do not any more. Partly because
    they were probably anyway only considered experimental, partly it
    were features that I'd suspect are not commonly used.

    Off the top of my head one was the support of ksh's alarm timers,

    The undocumented 'alarm' built-in was removed in 93u+m/1.0 because it was broken and could easily crash the shell; Korn kept it in for years but never fixed it. My standards are higher than that. I prefer not to ship stuff that I know is broken.

    It is still present on the dev branch which will eventually become 93u+m/1.1, though. If we manage to fix it properly, then it will reappear in that
    release, but I can't promise it.

    and another thing was in context of discipline functions used on
    shell variables. (In the latter case I have no certainty whether
    it's a shell issue or whether I should have done this differently.

    An example for the above mentioned effect shows up like that...

    $ pushftp
    pushftp: LINES.set: line 44: .kosta.rows=24: no parent
    pushftp: COLUMNS.set: line 45: .kosta.cols=79: no parent
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]
    [...]
    I can't tell whether I have missed something with discipline functs
    on built-in variables, how I used the namespaces, or something like
    that. (My tries to fix that in my program logic all failed.) Though
    the above "no parent" error messages did not show up with ksh93u+.

    The fix for this is a prior 'typeset .kosta' (or even just '.kosta=') to
    create the parent variable.

    To the best of my knowledge, this is exactly the same in 93u+ 2012-08-01:

    $ /bin/ksh -c 'echo ${.sh.version}; .kosta.rows=24'
    Version AJM 93u+ 2012-08-01
    /bin/ksh: .kosta.rows=24: no parent
    $ /bin/ksh -c 'typeset .kosta; .kosta.rows=24; echo ${.kosta.rows}'
    24

    --
    || modernish -- harness the shell
    || https://github.com/modernish/modernish
    ||
    || KornShell lives!
    || https://github.com/ksh93/ksh

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Martijn Dekker on Tue Jul 2 10:52:39 2024
    Thanks for your post, Martijn!

    On 02.07.2024 05:35, Martijn Dekker wrote:
    Op 22-06-2024 om 13:15 schreef Janis Papanagnou:
    [...]

    An example for the above mentioned effect shows up like that...

    $ pushftp
    pushftp: LINES.set: line 44: .kosta.rows=24: no parent
    pushftp: COLUMNS.set: line 45: .kosta.cols=79: no parent
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]
    [...]
    I can't tell whether I have missed something with discipline functs
    on built-in variables, how I used the namespaces, or something like
    that. (My tries to fix that in my program logic all failed.) Though
    the above "no parent" error messages did not show up with ksh93u+.

    The fix for this is a prior 'typeset .kosta' (or even just '.kosta=') to create the parent variable.

    Adding this typeset doesn't change the observed behavior
    with my test case (neither in u+ nor in u+m).


    To the best of my knowledge, this is exactly the same in 93u+ 2012-08-01:

    $ /bin/ksh -c 'echo ${.sh.version}; .kosta.rows=24'
    Version AJM 93u+ 2012-08-01
    /bin/ksh: .kosta.rows=24: no parent

    A plain assignment to a namespace variable doesn't trigger
    the "no parent" error in my environment.

    $ /bin/ksh -c 'typeset .kosta; .kosta.rows=24; echo ${.kosta.rows}'
    24


    But calling any script does; two examples

    (1000)$ pushftp # called in my ksh93u+m shell context
    pushftp: LINES.set: line 46: .kosta.rows=24: no parent
    pushftp: COLUMNS.set: line 47: .kosta.cols=80: no parent
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]
    (1001)$ ksh93u+
    (1002)$ pushftp
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]
    (1003)$ # here I'm leaving ksh93u+ with ^D
    (1003)$ pushftp # again in ksh93u+m
    pushftp: LINES.set: line 46: .kosta.rows=24: no parent
    pushftp: COLUMNS.set: line 47: .kosta.cols=80: no parent
    pushftp: line 11: 1: pushftp file ftp-path [ domain ]
    (1004)$ Version AJM 93u+m/1.0.8 2024-01-01 # Esc ^V

    (1004)$ isodate # 'isodate' contains: date "+%F" "$@"
    isodate: LINES.set: line 46: .kosta.rows=24: no parent
    isodate: COLUMNS.set: line 47: .kosta.cols=80: no parent
    2024-07-02
    (1005)$ ksh93u+
    (1006)$ isodate
    2024-07-02
    (1007)$

    I have no idea why there's a difference between u+ and u+m here.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)