From Newsgroup: comp.unix.shell

On 23.01.2025 23:06, Kaz Kylheku wrote:

Which which is which? Burn the which!

:-)

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 23.01.2025 23:46, Keith Thompson wrote:

Kaz Kylheku <643-408-1753@kylheku.com> writes:
[...]

The obvious situation is double quotes. Inside double quotes, parameter
expansion happens, but not tilde expansion (not to mention pathname
expansion (globbing) and perhaps some other things).

So this won't work:

PATH="$PATH:~/bin"

Sorry I don't have details, but it is true nonetheless.

There they are.

[...]

In fact it probably will *partially* work. As I mentioned elsethread,
bash expands a leading ~ (or even ~username) in an element of $PATH when executing a command.

Interpretation of any _leading_ '~' is the documented behavior for
tilde expansion.

But if you run a program from the command line that invokes another
program (say, a C program that calls system()), it won't treat that
element of $PATH the same way.

I think the point is that the _shell_ expands the expression.

Whether you do PATH="~" or PATH='~' or PATH=\~ , in all cases you have
defined a variable value that contains '~', a tilde. Not an expanded tilde-expression. So if you 'exec' a command that PATH variable (with
the literal tilde) will get passed to the executable environment (and
no shell interpretation takes place).

Use PATH=~ instead if you want it expanded.

For this and other reasons, though you *can* have a literal ~ in $PATH
in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Janis

A literal '$HOME'
won't work at all, but that's less likely to be a problem if your at all aware of how double quotes work in the shell.

I suggest that bash's undocumented behavior is less than helpful.
I'll probably submit a bug report.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vmvp3d$2671i$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 23:46, Keith Thompson wrote:

[snip]
For this and other reasons, though you *can* have a literal ~ in $PATH
in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Or just don't use it, and then you don't have to worry about it.

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Kenny McCormack wrote:

In article <ccr96l-eot.ln1@ID-313840.user.individual.net>,
Geoff Clare <netnews@gclare.org.uk> wrote:
...

Yes. Perhaps I trimmed too much. The post I was replying to said >>"$HOME/bin [..] is better than ~/bin, because tilde expansion is not, >>AFAIK, included in POSIX" and $HOME is also, of course, expanded before >>PATH is assigned. So there is no reason to prefer $HOME/bin over ~/bin >>since (when used in an assignment) they are equivalent in POSIX.

1) I have no idea what your beef with Kaz is. It seems silly at best.

No beef. Quite the opposite in fact - I was attempting to be apologetic
for having written something that could be misinterpreted (because I
trimmed too much).

2) I know this isn't going to sit well with you, but there absolutely are situations (in bash) where ~ doesn't work as a substitute for $HOME, when setting the various "path" variables. I know that I had lines in .profile and/or .bashrc like: PATH=~/bin:$PATH (and similar) that did not work (that is, the value stored in the variable contained an explicit tilde rather
than the expanded value - and this, of course, doesn't work at runtime). Replacing the tilde with $HOME fixes this.

Sorry I don't have details, but it is true nonetheless.

As others have already pointed out, it's probably because you quoted the
tilde. Which means when I carefully put the caveat "when used in an assignment" into what I wrote about equivalence of $HOME/bin and ~/bin,
I wasn't careful enough - I should have written "when used in an
unquoted assignment".
--
Geoff Clare <netnews@gclare.org.uk>
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vmu94j$1q2lp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 17:47, Dan Cross wrote:

[snip]

: term; echo $HOME
/home/cross
: term; pwd
/home/cross
: term; echo echo xyzzy >bin/quux
: term; chmod +x bin/quux
: term; which quux
/home/cross/bin/quux
: term; quux
xyzzy
: term; exec ksh
: term; export PATH=~/bin:/bin:/usr/bin
: term; echo $PATH
/home/cross/bin:/bin:/usr/bin
: term; which quux
/home/cross/bin/quux
: term; quux
xyzzy
: term; export PATH="$HOME/bin:/bin:/usr/bin"
: term; echo $PATH
/home/cross/bin:/bin:/usr/bin
: term; which quux
/home/cross/bin/quux
: term; quux
xyzzy
: term; export PATH="~/bin:/bin:/usr/bin"
: term; echo $PATH
~/bin:/bin:/usr/bin
: term; which quux
: term; quux
ksh: quux: not found
: term; exec /usr/pkg/bin/bash
: term; echo $PATH
~/bin:/bin:/usr/bin
: term; which quux
: term; quux
xyzzy
: term;

Bash behaves strange here; 'which' doesn't find the executable but >nonetheless bash executes it, shows its output?

Bash doesn't behave strangely here at all. Outside of its POSIX
mode, it's free to implement whatever behavior it likes with
respect to `~` expansion, and this is the behavior the author
has chosen.

But, critically, that does not mean that the programs _that it
invokes_ have to do the same. So while `bash` can perform `~`
expansion on the components of $PATH when it's searching for a
program to execute, one shouldn't be surprised if other programs
don't do the same thing.

As was pointed out, `which` is not built into the shell, and
therefore, not obligated to follow `bash`'s expansion rules.

I've changed the above test frame to remove some unnecessary parts
to eliminate distracting complexity. The output for three shells:

Sorry, I think this cuts too much: the point was to show how the
different shells handle `~` in different contexts; the actual
strings that are in `$PATH` are important to show.

[snip]

I suppose the moral is, for maximal portability, prefer using
$HOME (or another similar variable) to `~` when setting $PATH
unless you're doing so in a context where you are sure that `~`
will be expanded during assignment.

IMO, the consequence is to not quote a tilde expression in the first
place if you want it expanded in a shell variable. *Violating* that
rule will _in Bash_ produce this strange effect (in the given setup).

To me it appears that not finding (by 'which') an executable but
executing it nonetheless qualifies as a bug [in Bash].

Not a bug, but yes, surprising.

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <4chc6l-lju.ln1@ID-313840.user.individual.net>,
Geoff Clare <netnews@gclare.org.uk> wrote:
...

As others have already pointed out, it's probably because you quoted the >tilde.

No, I didn't.
--
Most Southerners interest in, knowledge of, and participation in politics begins with
and ends with: Screw the blacks. If a guy is onboard with that, he's our guy!

Get them back in chains where they belong!
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 14:46, Dan Cross wrote:

In article <vmu94j$1q2lp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Bash behaves strange here; 'which' doesn't find the executable but
nonetheless bash executes it, shows its output?

Bash doesn't behave strangely here at all. Outside of its POSIX
mode, it's free to implement whatever behavior it likes with
respect to `~` expansion, and this is the behavior the author
has chosen.

But, critically, that does not mean that the programs _that it
invokes_ have to do the same. So while `bash` can perform `~`
expansion on the components of $PATH when it's searching for a
program to execute, one shouldn't be surprised if other programs
don't do the same thing.

As I see it, Bash does the same correct assignment to PATH in
case the tilde-expression is quoted or unquoted; here there's
no difference. Quoted or escaped it's a literal tilde in PATH,
unquoted it's an expanded tilde-expression. (As in all those
shells; bash, ksh, zsh, dash, sh.)

As was pointed out, `which` is not built into the shell, and
therefore, not obligated to follow `bash`'s expansion rules.

And this (at least) seems to be the source of an inconsistency;
'which' is also in other shells not a built-in. But all other
shells I tested (ksh, zsh, dash, sh) handle it consistently;
if 'which' ("/usr/bin/which") detects no program [in PATH] it
should not execute some program. Other shells do that correctly.

I've changed the above test frame to remove some unnecessary parts
to eliminate distracting complexity. The output for three shells:

Sorry, I think this cuts too much: the point was to show how the
different shells handle `~` in different contexts; the actual
strings that are in `$PATH` are important to show.

The actual strings and their handling WRT PATH is actually the
same in all these shells! Those test-cases have hidden the real
problem, so I reduced it to show that it's not (not per se) the
"tilde in PATH" that is the problem; Bash (as the other shells)
correctly expands an unquoted tilde-expression, and correctly
leaves it in PATH literally if it's quoted or escaped.

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

It appears as if Bash invokes a tilde-expansion twice(!); once
[if unquoted] when the assignment happens, and another time
when the path-search is actually done. Both of our test-cases
indicated that Bash seems to tilde-expand a PATH variable value
a second time; so while 'which' (as other shells and programs)
will not find any executable in "./~/bin/" (which would be the
correct interpretation of a quoted "~/bin") Bash does.

That inconsistency - and deviating from all the other shells -,
if not a bug, looks like a Bad Design Idea if it's been done
deliberately. - YMMV.

Janis

[...]

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 14:33, Dan Cross wrote:

In article <vmvp3d$2671i$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 23:46, Keith Thompson wrote:

[snip]
For this and other reasons, though you *can* have a literal ~ in $PATH
in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Or just don't use it, and then you don't have to worry about it.

But as a Ksh (or any non-Bash shell) user I don't have
to worry about it. (Why shall I see any issue with it?)
(But as I very early already said; I anyway don't use
tilde-expressions in scripts, even as a Ksh user.)

But more importantly; shell programmers shall be well aware
of what quotes mean in shells! They are not just fancy things
or accessories that one may or may not use as one likes. They
have clear semantics and are essential in shell programming.

If you want tilde-expressions expanded _don't quote them_.
It's not much different from file-globbing; don't escape or
quote a '*' (or other globbing meta-characters) if you want
it to become expanded.

The suggestion to "not use" this ~ or that * is misguiding.
Know the shell concepts! - Or know your shell, at least,
with all its inconsistencies and/or (where applicable) bugs.

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Kaz Kylheku <643-408-1753@kylheku.com> wrote:

On 2025-01-23, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Bash behaves strange here; 'which' doesn't find the executable but
nonetheless bash executes it, shows its output?

which is a nonstandard command; the POSIX command is type.

In those systems where "which" exists at all, it is often
a locally brewed program that is not exactly the same like
the one in other systems.

In Debian and derivatives thereof, /usr/bin/which is a shell script.

I see that in MacOS, there is a /usr/bin/which whose --help
says to send mail to which-bugs<at>gnu.org; the man page
implicates a Carlo Wood as the culprit behind it.
It doesn't appear to be part of GNU Coreutils.

Which which is which? Burn the which!

Slackware has the binary which and the manpage attributes
Carlo Wood at gnu.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vn0bpf$29qe6$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 24.01.2025 14:46, Dan Cross wrote:

In article <vmu94j$1q2lp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Bash behaves strange here; 'which' doesn't find the executable but
nonetheless bash executes it, shows its output?

Bash doesn't behave strangely here at all. Outside of its POSIX
mode, it's free to implement whatever behavior it likes with
respect to `~` expansion, and this is the behavior the author
has chosen.

But, critically, that does not mean that the programs _that it
invokes_ have to do the same. So while `bash` can perform `~`
expansion on the components of $PATH when it's searching for a
program to execute, one shouldn't be surprised if other programs
don't do the same thing.

As I see it, Bash does the same correct assignment to PATH in
case the tilde-expression is quoted or unquoted; here there's
no difference.

Bash demonstrably does not assign the same thing to $PATH when
the `~` is quoted or escaped versus unquoted/unescaped:

```
: term; exec bash
: term; PATH="~/bin:/bin:/usr/bin"
: term; echo $PATH
~/bin:/bin:/usr/bin
: term; PATH=~/bin:/bin:/usr/bin
: term; echo $PATH
/home/cross/bin:/bin:/usr/bin
: term;
```

Note that in the first form, the literal `~` character is in
$PATH, while in the second it has been expanded, and the result
is a pathname relative to my home directory.

Quoted or escaped it's a literal tilde in PATH,
unquoted it's an expanded tilde-expression. (As in all those
shells; bash, ksh, zsh, dash, sh.)

Yes, is what was shown.

As was pointed out, `which` is not built into the shell, and
therefore, not obligated to follow `bash`'s expansion rules.

And this (at least) seems to be the source of an inconsistency;
'which' is also in other shells not a built-in. But all other
shells I tested (ksh, zsh, dash, sh) handle it consistently;

If it's not built into those shells, then those shells don't
"handle" `which` in any meaningful way, other than exec'ing it
as they would any other program.

But `which` was just an example of a program that's not built
into the shell that inspects $PATH, which it inherited from the
shell that invoked it.

The interesting point of the example was to show that, as it
goes to run a program, `bash` will interpret each component of
`$PATH` and perform `~` expansion, but since it doesn't modify
`$PATH` when it does so, the programs that it invokes (and that
inherit `$PATH` from it) may not find programs that `bash` does
if they use execlp/execvp/etc, or even just inspect $PATH
directly.

if 'which' ("/usr/bin/which") detects no program [in PATH] it
should not execute some program. Other shells do that correctly.

Why do you say this? As I wrote earlier, nothing prevents
`bash` from treating $PATH however it likes outside of its POSIX
mode, and its behavior is entirely correct according to the
rules by which it does things. If other shells do things
differently, then that's their behavior; bash need not be bound
by it.

I've changed the above test frame to remove some unnecessary parts
to eliminate distracting complexity. The output for three shells:

Sorry, I think this cuts too much: the point was to show how the
different shells handle `~` in different contexts; the actual
strings that are in `$PATH` are important to show.

The actual strings and their handling WRT PATH is actually the
same in all these shells!

No it's not; that was the point.

Those test-cases have hidden the real problem,

That's the thing; there is no "problem." There are some
differences between how shells behave, but they're different
programs, so why is that bad?

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

I don't see any reason why that _must_ be true. Any given shell
is free to interpret $PATH any way it choses, or even use a
variable named something other than `PATH` for the search path.
For example, the `rc` shell used on 10th Edition Research Unix
and Plan 9 doesn't used `$path` (lower-case) to find programs.

It appears as if Bash invokes a tilde-expansion twice(!); once
[if unquoted] when the assignment happens, and another time
when the path-search is actually done.

Yes. That's precisely what I was showing.

Both of our test-cases
indicated that Bash seems to tilde-expand a PATH variable value
a second time;

I fail to see where this is being done a "second time": it seems
more likly that `bash` doesn't make any special note of the `~`
in `PATH="~/bin:whatever"` until it actuall goes to run a
program.

so while 'which' (as other shells and programs)
will not find any executable in "./~/bin/" (which would be the
correct interpretation of a quoted "~/bin") Bash does.

You seem to have invented a definition of "correct" here and are
pursuing it aggressively, but that is just one definition from
a large set of such definitions; there's no reason to assume the
behavior you expect here is any more correct than what `bash`
does.

That inconsistency - and deviating from all the other shells -,

Careful: "all other shells" is a pretty big net, and it wouldn't
surprise me at all if this broke down if you expanded the set of
things you were looking at. If you really want to twist your
nogging, have a look at what `csh` does, for example.

if not a bug, looks like a Bad Design Idea if it's been done
deliberately. - YMMV.

It's obviously being done deliberately, and is easy to find in
the source code:
https://git.savannah.gnu.org/cgit/bash.git/tree/findcmd.c#n533

You may consider it bad design, and you're well within your
rights to do so, but opinions on that vary, and it doesn't mean
yours is correct.

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vn0cno$29vrs$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 24.01.2025 14:33, Dan Cross wrote:

In article <vmvp3d$2671i$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 23:46, Keith Thompson wrote:

[snip]
For this and other reasons, though you *can* have a literal ~ in $PATH >>>> in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Or just don't use it, and then you don't have to worry about it.

But as a Ksh (or any non-Bash shell) user I don't have
to worry about it. (Why shall I see any issue with it?)

Because you might want to put whatever you assign to `PATH`
in quotes, for instance if their are spaces in one of the
component pathnames (people run `bash` on windows and all
kinds of weird places) and the behavior differs. $HOME is
pleasantly boring by comparison.

[snip]
But more importantly; shell programmers shall be well aware
of what quotes mean in shells! They are not just fancy things
or accessories that one may or may not use as one likes. They
have clear semantics and are essential in shell programming.

If you want tilde-expressions expanded _don't quote them_.

What if the expression refers to a file name with a space in
it? Of course, one can escape the whitespace characters in
filenames, but that gets tedious.

It's not much different from file-globbing; don't escape or
quote a '*' (or other globbing meta-characters) if you want
it to become expanded.

The suggestion to "not use" this ~ or that * is misguiding.
Know the shell concepts! - Or know your shell, at least,
with all its inconsistencies and/or (where applicable) bugs.

See above.

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 23.01.2025 22:50, Keith Thompson wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

[...]

To me it appears that not finding (by 'which') an executable but
executing it nonetheless qualifies as a bug [in Bash].

I wouldn't call it a bug in bash. Rather, it's a mismatch between bash
and which.

I'm not sure I can follow you what you mean by "a mismatch". Kornshell,
for example, also accesses /usr/bin/which (but it behaves correctly).

/usr/bin/which is not tied to any shell. Kornshell accesses
"/usr/bin/which" if you type "which" or "/usr/bin/which" at the
prompt -- like any other external command.

From the which(1) man page (emphasis added):

which returns the pathnames of the files (or links) which would be
executed in the current environment, had its arguments been given as
commands **in a strictly POSIX-conformant shell**. It does this by
searching the PATH for executable files matching the names of the
arguments. It does not canonicalize path names.

bash, unless it's in POSIX mode, is not a strictly POSIX-conformant
shell. The feature is disabled when bash is in POSIX mode. It's to be expected that "which" doesn't know about this.

One could argue that bash's behavior is a legitimate and useful
extension. If so, this is a mismatch between bash and which. There's
nothing inherently wrong with bash having non-POSIX extensions,
particularly since bash has a "--posix" option to disable them.

But since the feature is undocumented and can quietly lead to
inconsistent behavior, I would actually argue that it's a bug in bash,
or at least a misfeature. (It turns out that bash has had this feature
from its earliest days.)

[...]
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 14:33, Dan Cross wrote:

In article <vmvp3d$2671i$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 23:46, Keith Thompson wrote:

[snip]
For this and other reasons, though you *can* have a literal ~ in $PATH >>>> in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Or just don't use it, and then you don't have to worry about it.

But as a Ksh (or any non-Bash shell) user I don't have
to worry about it. (Why shall I see any issue with it?)
(But as I very early already said; I anyway don't use
tilde-expressions in scripts, even as a Ksh user.)

But more importantly; shell programmers shall be well aware
of what quotes mean in shells! They are not just fancy things
or accessories that one may or may not use as one likes. They
have clear semantics and are essential in shell programming.

If you want tilde-expressions expanded _don't quote them_.
It's not much different from file-globbing; don't escape or
quote a '*' (or other globbing meta-characters) if you want
it to become expanded.

The suggestion to "not use" this ~ or that * is misguiding.
Know the shell concepts! - Or know your shell, at least,
with all its inconsistencies and/or (where applicable) bugs.

~ is equivalent to $HOME in most contexts. I suggest that assuming that
~, like $HOME, is expanded in double quotes is a very easy mistake to
make. And the bash misfeature we're discussing can make it hard to
detect that mistake.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 14:46, Dan Cross wrote:

[...]

As was pointed out, `which` is not built into the shell, and
therefore, not obligated to follow `bash`'s expansion rules.

And this (at least) seems to be the source of an inconsistency;
'which' is also in other shells not a built-in. But all other
shells I tested (ksh, zsh, dash, sh) handle it consistently;
if 'which' ("/usr/bin/which") detects no program [in PATH] it
should not execute some program. Other shells do that correctly.

If by "correctly" you mean "POSIXly", you're right, but that's not the
only possible meaning of the word.

/usr/bin/which is limited in what it can do. It follows POSIX-specified behavior for $PATH; it doesn't recognize any shell-specific rules.

In particular, "which" does not handle:

- aliases
- shell functions
- shell builtin commands
- shell keywords (try "which if")
- Bash-specific behavior for literal '~' in $PATH

The behavior of "which" when searching in $PATH does match the behavior
of bash *in POSIX mode*.

I do agree that bash's non-POSIX behavior is a misfeature, possibly a
bug, but because it can be invoked accidentally and cause inconsistent
behavior when invoking commands directly vs. indirectly, not because it
doesn't match the behavior of /usr/bin/which. (And because it's
undocumented.)

[...]

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

What do you mean by "shall result?

All shells that conform to POSIX behave as you describe. bash doesn't
conform to POSIX unless you ask it to. Neither do csh, tcsh, and fish.

It appears as if Bash invokes a tilde-expansion twice(!); once
[if unquoted] when the assignment happens, and another time
when the path-search is actually done. Both of our test-cases
indicated that Bash seems to tilde-expand a PATH variable value
a second time; so while 'which' (as other shells and programs)
will not find any executable in "./~/bin/" (which would be the
correct interpretation of a quoted "~/bin") Bash does.

Bash does something similar to tilde-expansion when searching $PATH for
a command, but tilde-expansion involves replacement of words on the
command line, whereas its treatment of literal ~ in $PATH happens
internally. This is admittedly a minor quibble.

BTW, it hadn't occurred to me that you can have a relative path in a
component of $PATH, but it does seem to work. I won't be taking
advantage of this information.

[...]
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 22:34, Dan Cross wrote:

In article <vn0bpf$29qe6$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 24.01.2025 14:46, Dan Cross wrote:

In article <vmu94j$1q2lp$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

Bash behaves strange here; 'which' doesn't find the executable but
nonetheless bash executes it, shows its output?

Bash doesn't behave strangely here at all. Outside of its POSIX
mode, it's free to implement whatever behavior it likes with
respect to `~` expansion, and this is the behavior the author
has chosen.

But, critically, that does not mean that the programs _that it
invokes_ have to do the same. So while `bash` can perform `~`
expansion on the components of $PATH when it's searching for a
program to execute, one shouldn't be surprised if other programs
don't do the same thing.

As I see it, Bash does the same correct assignment to PATH in
case the tilde-expression is quoted or unquoted; here there's
no difference.

Bash demonstrably does not assign the same thing to $PATH when
the `~` is quoted or escaped versus unquoted/unescaped: [...]

What I tried to express was that Bash does the same assignment
as the other shells; in both cases. In the quoted case it takes
it literally, and in the unquoted case it expands it. There's no
difference here. The assignments are all handled _consistently_
across all the shells including Bash.

As was pointed out, `which` is not built into the shell, and
therefore, not obligated to follow `bash`'s expansion rules.

And this (at least) seems to be the source of an inconsistency;
'which' is also in other shells not a built-in. But all other
shells I tested (ksh, zsh, dash, sh) handle it consistently;

If it's not built into those shells, then those shells don't
"handle" `which` in any meaningful way, other than exec'ing it
as they would any other program.

The point is; 'which' should consider PATH, and Bash should
consider PATH. All shells (but Bash) do that in a consistent
way.

Depending on how the assignment had been made, literal tilde
or expanded tilde expression, the PATH value should be taken
as it is, IMO, and not tilde-expanded a second time.

This is nothing but a "creative" implementation in Bash; one
that is unnecessary, one that differs from the other shells,
and one that creates (IMO) inconsistent behavior, behavior
that lead to fancy suggestions to _not use_ tilde in PATH.

But `which` was just an example of a program that's not built
into the shell that inspects $PATH, which it inherited from the
shell that invoked it.

The interesting point of the example was to show that, [...]

I acknowledge and had acknowledged your test sample before.
(I just rewrite it for me to streamline the calls, isolate
and better see the source of the issue)

if 'which' ("/usr/bin/which") detects no program [in PATH] it
should not execute some program. Other shells do that correctly.

Why do you say this? As I wrote earlier, nothing prevents
`bash` from treating $PATH however it likes outside of its POSIX
mode, and its behavior is entirely correct according to the
rules by which it does things. If other shells do things
differently, then that's their behavior; bash need not be bound
by it.

Of course one can implement ones shell as he likes. As said
above, I like shells with a consistent implementation; where
a negative match of 'which' (or 'type' or 'whence') [for any
given program] should also result in the shells to not find
and execute it.

If the goal of your statement is just to say that implementers
can do what they like, I can agree. (But that's a rather weak
statement, IMO, and not very useful when valuating shells and
any [IMO] inconsistencies.) And if you think that my used term
"bug" for that is inappropriate, I can live with that as well.

[...]

The actual strings and their handling WRT PATH is actually the
same in all these shells!

No it's not; that was the point.

Above we agreed that the _variable_ assignment and expansion
of PATH is the same (in all shells). The difference [in Bash]
appears only when Bash does a PATH search to find appropriate
commands. My point was that this _path search_ does a second
expansion.

Those test-cases have hidden the real problem,

That's the thing; there is no "problem." There are some
differences between how shells behave, but they're different
programs, so why is that bad?

Because it's inconsistent, it behavior is unexpected, it does
in a very subtle way surprise the users, and folks that know
this inconsistency advise to thus better not use '~' in the
first place; to me that's all clear factors why that is "bad".

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

I don't see any reason why that _must_ be true.

Well, we see that shells handle escaping/quoting of variables
in a _consistent_ way. Why should I believe that a shell value
with a tilde should be handled inconsistently?

If you set PATH=$HOME or PATH=~ all works smoothly. If you use
PATH='$HOME' or PATH='~' then these path components are taken
literally. But [in Bash] only in case of '$HOME'.

Any given shell is free to interpret $PATH any way it choses, [...]

(We had covered that already.)

It appears as if Bash invokes a tilde-expansion twice(!); once
[if unquoted] when the assignment happens, and another time
when the path-search is actually done.

Yes. That's precisely what I was showing.

Both of our test-cases
indicated that Bash seems to tilde-expand a PATH variable value
a second time;

I fail to see where this is being done a "second time": it seems
more likly that `bash` doesn't make any special note of the `~`
in `PATH="~/bin:whatever"` until it actuall goes to run a
program.

$ PATH='~/bin:/usr/bin:/bin' bash -c 'echo $PATH'
~/bin:/usr/bin:/bin

$ PATH='~/bin:/usr/bin:/bin' bash -c 'echo $PATH; quux'
~/bin:/usr/bin:/bin
xyzzy - ~/bin:/usr/bin:/bin

$ cat /home/jap/bin/quux
echo xyzzy - "$PATH"

so while 'which' (as other shells and programs)
will not find any executable in "./~/bin/" (which would be the
correct interpretation of a quoted "~/bin") Bash does.

You seem to have invented a definition of "correct" here and are
pursuing it aggressively, but that is just one definition from
a large set of such definitions; there's no reason to assume the
behavior you expect here is any more correct than what `bash`
does.

The call it "consistent" vs. "inconsistent". (Modulo what we
have already said above.)

That inconsistency - and deviating from all the other shells -,

Careful: "all other shells" is a pretty big net, and it wouldn't
surprise me at all if this broke down if you expanded the set of
things you were looking at. If you really want to twist your
nogging, have a look at what `csh` does, for example.

No, I will not compare the shells sensibly used for programming
with Csh.

I was aware that if I write "all other shells" that someone will
be pricking. I tried with the common shells (Ksh, Bash, Zsh, Sh,
Dash); I think this is a sensible set of shells to discuss things.

if not a bug, looks like a Bad Design Idea if it's been done
deliberately. - YMMV.

It's obviously being done deliberately, and is easy to find in
the source code: https://git.savannah.gnu.org/cgit/bash.git/tree/findcmd.c#n533

You may consider it bad design, and you're well within your
rights to do so, but opinions on that vary, and it doesn't mean
yours is correct.

So you think that behavior of Bash is good design here? - Okay,
noted.

Janis

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 23:00, Keith Thompson wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 14:46, Dan Cross wrote:

[...]

/usr/bin/which is limited in what it can do. It follows POSIX-specified behavior for $PATH; it doesn't recognize any shell-specific rules. [...]

Sure.

[...]

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

What do you mean by "shall result?

I mean that a shell should behave consistently. (I think Bash does
not in the given case.)

All shells that conform to POSIX behave as you describe. bash doesn't conform to POSIX unless you ask it to. Neither do csh, tcsh, and fish.

We were speaking about shell programming, so [seriously] I don't
consider Csh and Tcsh as sensible sample shells for the discussion.
(Thinking about it, I wonder whether Bash inherited tilde-handling
from Csh, maybe; that would at least explain something.)

(I don't know Fish, so I cannot comment on that.)

[...]

BTW, it hadn't occurred to me that you can have a relative path in a component of $PATH, but it does seem to work. I won't be taking
advantage of this information.

Yes. Some prefer to add '.' to PATH. (Though I have no intention
to discuss that habit.)

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 22:38, Dan Cross wrote:

In article <vn0cno$29vrs$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 24.01.2025 14:33, Dan Cross wrote:

In article <vmvp3d$2671i$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

On 23.01.2025 23:46, Keith Thompson wrote:

[snip]
For this and other reasons, though you *can* have a literal ~ in $PATH >>>>> in bash, it's best to avoid it and use $HOME instead.

Or use it correctly, unquoted and unescaped.

Or just don't use it, and then you don't have to worry about it.

But as a Ksh (or any non-Bash shell) user I don't have
to worry about it. (Why shall I see any issue with it?)

Because you might want to put whatever you assign to `PATH`
in quotes, for instance if their are spaces in one of the
component pathnames (people run `bash` on windows and all
kinds of weird places) and the behavior differs. $HOME is
pleasantly boring by comparison.

Sure. That's why you can write, say,

PROJECT_PATH="Yet another space odyssey"
PATH=~/bin:"My Windows Bin":$PROJECT_PATH:'/tmp/$LITERAL':.

(The standard way. Clean unambiguous quoting. No issues.)

[snip]
But more importantly; shell programmers shall be well aware
of what quotes mean in shells! They are not just fancy things
or accessories that one may or may not use as one likes. They
have clear semantics and are essential in shell programming.

If you want tilde-expressions expanded _don't quote them_.

What if the expression refers to a file name with a space in
it? Of course, one can escape the whitespace characters in
filenames, but that gets tedious.

The (standard-)rules are simple and clean; see my sample above.

To say it explicitly; values in variables are *not* subject to
another word-splitting in assignments [in standard shells].

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 24.01.2025 22:42, Keith Thompson wrote:

~ is equivalent to $HOME in most contexts. I suggest that assuming that
~, like $HOME, is expanded in double quotes is a very easy mistake to
make. And the bash misfeature we're discussing can make it hard to
detect that mistake.

Yes, to both.

(It certainly helps to know the standard shell quoting mechanisms.
That's so basic that it cannot be overestimated and should always
be emphasized. - I've experienced that even regular, experienced
shell users don't seem to know the exact mechanism in all cases.)

Anyone reading the docs? :-)

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vn2hsj$2pe96$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

[snip]

Most of this discussion seems to be talking at cross-purposes.
I don't see much point in responding to the specifics.

My point was simply to show that shell behavior varies with
respect to how $PATH is treated; caveat emptor.

I don't usually use bash, but if you do and you really don't
like this behavior you can turn it off.

You may consider it bad design, and you're well within your
rights to do so, but opinions on that vary, and it doesn't mean
yours is correct.

So you think that behavior of Bash is good design here? - Okay,
noted.

Please don't put words in my mouth. I don't really have much of
an opinion on whether what bash does here is good or not: it is
neither wrong nor right, it simply is what it is.

I do recognize that, for better or worse, the universe of useful
or even just interesting software is far larger than the small
sample set you seem to want to constrain it to. In that very
large space, there have been, and will continue to be, many
different design points and implementations with respect to how
these sorts of things are handled; indeed, just a couple of
years ago shells were an active area of research: https://nikos.vasilak.is/p/pash:hotos:2021.pdf

I have found that generally one would be wise not to make too
many assumptions about what is "correct" versus what one is just
most familiar/comfortable with.

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

cross@spitfire.i.gajendra.net (Dan Cross) writes:

In article <vn0bpf$29qe6$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

[...]

if not a bug, looks like a Bad Design Idea if it's been done
deliberately. - YMMV.

It's obviously being done deliberately, and is easy to find in
the source code: https://git.savannah.gnu.org/cgit/bash.git/tree/findcmd.c#n533

The code in question is:

xpath = (posixly_correct == 0 && *path == '~') ? bash_tilde_expand (path, 0) : path;

where `path` is an element of $PATH.

This is obviously deliberate, and I see similar code (without the posixly_correct condition) in bash 1.05 from 1990.

According to Wikipedia, POSIX began in 1988, and the initial
release of bash was in 1989, just a year later. Obviously the
authors thought that expanding literal '~'s in $PATH was a good
idea at the time, and it's not suprising that they didn't pay much
attention to POSIX. It would have been nice if they'd documented it.

You may consider it bad design, and you're well within your
rights to do so, but opinions on that vary, and it doesn't mean
yours is correct.

As far as I can tell (I haven't investigated), all Bourne-based
shells other than Bash treat $PATH in the same way when executing
commands. Bash is the only shell I'm aware of that expands literal
'~' in $PATH.

The fact that it's not POSIX compliant behavior is at worst mildly
annoying. Bash correctly disables it in POSIX mode (enabled by the
--posix option, by running "set -o posix", by setting $POSIXLY_CORRECT
before starting bash, or by invoking it sas "sh").

I dislike the behavior because (a) it can quietly cause unexpected
problems (it's easy to miss the fact that $HOME is expanded within
double quotes and ~ is not), and (b) it's not documented. I certainly
consider it a misfeature. Whether it rises to the level of a bug is
a matter of opinion.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

cross@spitfire.i.gajendra.net (Dan Cross) writes:

In article <vn2hsj$2pe96$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

[snip]

Most of this discussion seems to be talking at cross-purposes.
I don't see much point in responding to the specifics.

My point was simply to show that shell behavior varies with
respect to how $PATH is treated; caveat emptor.

I don't usually use bash, but if you do and you really don't
like this behavior you can turn it off.

As far as I can tell, you can only turn off the behavior by
running in POSIX mode, which disables a lot of other Bash-specific functionality. Personally, I'm unwilling to do that. I think I'd
like to see a "set -o" setting that disables just this feature.

You can *avoid* it by being careful not to put literal '~' characters
in $PATH (specifically at the beginning of any element of $PATH).
That's what I do (unintentionally before now, deliberately now that
I know about it.)
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 23:00, Keith Thompson wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 14:46, Dan Cross wrote:

[...]

/usr/bin/which is limited in what it can do. It follows POSIX-specified
behavior for $PATH; it doesn't recognize any shell-specific rules. [...]

Sure.

[...]

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

What do you mean by "shall result?

I mean that a shell should behave consistently. (I think Bash does
not in the given case.)

Consistently with what? Bash consistently expands literal '~'s in
$PATH, and consistently disables that expansion in POSIX mode.

All shells have shell-specific features. What's odd about this case is
that bash has a POSIX-violating feature that affects command name
resolution.

If this particular feature were documented, I'd have less of a problem
with it (but I'd still avoid using it).

All shells that conform to POSIX behave as you describe. bash doesn't
conform to POSIX unless you ask it to. Neither do csh, tcsh, and fish.

I should have mentioned that csh, tcsh, and fish don't conform to POSIX
at all.

We were speaking about shell programming, so [seriously] I don't
consider Csh and Tcsh as sensible sample shells for the discussion.
(Thinking about it, I wonder whether Bash inherited tilde-handling
from Csh, maybe; that would at least explain something.)

No, csh and tcsh (at least in current versions) don't expand literal
'~'s in $PATH.

(I don't know Fish, so I cannot comment on that.)

Nor do I.

I don't mind excluding non-Bournish shells like csh, tcsh, and
fish from the discussion, but I like to mention now and then that
they're being excluded. They are shells, after all.

[...]

BTW, it hadn't occurred to me that you can have a relative path in a
component of $PATH, but it does seem to work. I won't be taking
advantage of this information.

Yes. Some prefer to add '.' to PATH. (Though I have no intention
to discuss that habit.)

Right, of course '.' is a relative path. I should have thought of that.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-01-26, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 23:00, Keith Thompson wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 14:46, Dan Cross wrote:

[...]

/usr/bin/which is limited in what it can do. It follows POSIX-specified >>> behavior for $PATH; it doesn't recognize any shell-specific rules. [...]

Sure.

[...]

The settings PATH=~/bin and PATH="~/bin" respectively shall
result in the same behavior across shells when searching for
programs; in the first case looking into "/home/someuser/bin/"
and in the second case looking into "./~/bin/" (i.e. a path
component with a local directory named "~").

What do you mean by "shall result?

I mean that a shell should behave consistently. (I think Bash does
not in the given case.)

Consistently with what? Bash consistently expands literal '~'s in
$PATH, and consistently disables that expansion in POSIX mode.

All shells have shell-specific features. What's odd about this case is
that bash has a POSIX-violating feature that affects command name
resolution.

It's a feature that (if used) leaks tildes into child processes via
the environment variable. Path resultion in child processes, if it
reaches a PATH element with a tilde, will somehow process that tilde.

I just tried this experiment. I made a directory named ~ and put ~:
as the leading element of PATH. I put a program called "foo" that
directory.

Surely enough, I can run "foo" from the parent directory above.

The exec functions treat ~ as an ordinary path component.

(I cannot do that out of Bash, which processes the tilde, but
the 'p' family of the exec functions will find it!)

This is a problem similar to "." being in PATH.

If someone has, say, "~/bin" in their PATH, ahead of /bin and /usr/bin,
I can put a malicious program in some directory called "~/bin"
somewhere in the filesystem, give that program the name of a common
external utility, and trick the user into changing into that location
where they will run this common command, resolving to my malicious
program.

If we regard this as a security hole, that atually raises the priority
and bolsters the argument that it ought to be removed even if it
breaks some users, perhaps through a process of noisy deprecation.

Furhermore, the case can be made that the exec stuff in the Linux kernel
or C libraries should be patched with a check against components with a
leading tilde.
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 26.01.2025 02:36, Keith Thompson wrote:

[...]
As far as I can tell (I haven't investigated), all Bourne-based
shells other than Bash treat $PATH in the same way when executing
commands. Bash is the only shell I'm aware of that expands literal
'~' in $PATH.

The fact that it's not POSIX compliant behavior is at worst mildly
annoying. Bash correctly disables it in POSIX mode (enabled by the
--posix option, by running "set -o posix", by setting $POSIXLY_CORRECT
before starting bash, or by invoking it sas "sh").

The problem I see with that is that you also disable other features,
sensible features, that are common in the prominent shells and that
you don't want to miss. - So Bash users, if they don't want to get
bitten in a subtle way, are probably advised to be pointed out to
that behavior, or, as suggested by others already, not to use tilde
at all with PATH in Bash.

I dislike the behavior because (a) it can quietly cause unexpected
problems (it's easy to miss the fact that $HOME is expanded within
double quotes and ~ is not), and (b) it's not documented. I certainly consider it a misfeature. [...]

The consequences are yet more fundamental; if you observe the behavior
of other quotings as well...

Whether PATH='~' or PATH="~", both are stored as literal value in all
shells (including Bash!). - That's fine.

Where PATH='$HOME' is (as expected) stored as a literal value in all
shells (also in Bash) and PATH="$HOME" is (as expected) expanded in
all shells (also in Bash) and store in expanded form. - That's also
fine.

All the variable storage and quoting is fine and consistent across
all shells (including Bash).

But in Bash a literal '$HOME' doesn't get expanded during path search.
And you see '~' in a PATH as being stored literally but nonetheless it
gets expanded in the path search. - In other words; the problem is not
the quoting.

That's IMO so stupid a design that I can only explain it with no one
having been courageous enough over the many decades to fix that old implementation. - Of course, the later you consider to fix such a
[mis-]design the harder you have it; development people are concerned
about software that is relying on that and thus reluctant to fix it.)

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 25.01.2025 16:35, Dan Cross wrote:

[...]

I don't usually use bash, but if you do and you really don't
like this behavior you can turn it off.

Neither do I use Bash. (Or rather; only rarely.)

[...]

Please don't put words in my mouth. [...]

Please accept my apology.

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 26.01.2025 03:07, Keith Thompson wrote:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 24.01.2025 23:00, Keith Thompson wrote:
[...]

I mean that a shell should behave consistently. (I think Bash does
not in the given case.)

Consistently with what? Bash consistently expands literal '~'s in
$PATH, and consistently disables that expansion in POSIX mode.

I think I have (already repeatedly) expanded on that, so let me just
enumerate some obvious aspects sparsely...
- expanding tilde once at dedicated places vs. twice
- consistency with other expansions (not done twice), e.g. '$HOME'
- (and if you accept that) consistency with the other shells

(Any single inconsistency is IMO already worthwhile to be critically discussed.)

Janis

[...]

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 26.01.2025 06:26, Kaz Kylheku wrote:

It's a feature that (if used) leaks tildes into child processes via
the environment variable. Path resultion in child processes, if it
reaches a PATH element with a tilde, will somehow process that tilde.

I just tried this experiment. I made a directory named ~ and put ~:
as the leading element of PATH. I put a program called "foo" that
directory.

Surely enough, I can run "foo" from the parent directory above.

The exec functions treat ~ as an ordinary path component.

(I cannot do that out of Bash, which processes the tilde, but
the 'p' family of the exec functions will find it!)

This is a problem similar to "." being in PATH.

[ Above context preserved for integrity. ]

If someone has, say, "~/bin" in their PATH, ahead of /bin and /usr/bin,
I can put a malicious program in some directory called "~/bin"
somewhere in the filesystem, give that program the name of a common
external utility, and trick the user into changing into that location
where they will run this common command, resolving to my malicious
program.

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.
So there would, IMO, not be a security hole (i.e. not because of that).

If we regard this as a security hole, that atually raises the priority
and bolsters the argument that it ought to be removed even if it
breaks some users, perhaps through a process of noisy deprecation.

Furhermore, the case can be made that the exec stuff in the Linux kernel
or C libraries should be patched with a check against components with a leading tilde.

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-01-26, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.
So there would, IMO, not be a security hole (i.e. not because of that).

The / is not part of a name; it is acting as the path component
separator. PATH allows relative paths with multiple components.
This is real!

$ PATH='~/bin':$PATH
$ cat '~/bin/cat' # script I prepared
#!/bin/sh
echo BOO!
$ cat # Bash in stock mode finds /bin/cat
^C
!130!
$ bash --posix # Bash in --posix mode falls victim
bash-4.4$ cat
BOO!
bash-4.4$

Someone who has literal ~/bin in their PATH so that their shell finds
utilities in their personal bin is about just as susceptible to an
attack as someone who has . in their PATH.

They might be even more susceptible. Here is why. Someone who has
~/bin in their PATH probably has a personal bin containing
custom programs with names distinct from standard utilities.
I have such things myself, such as "tagify".

The attack can be perpetrated using the name "tagify" regardless
of where the user has placed '~/bin' into PATH: first or last,
because that name is not found in any of the other locations
listed in PATH.

If we happen to have access to the user's ~/bin directory, we can set a
trap for each of the custom program names we find there.

Whereas the . exploit typically requires . to be ahead of
the standrad locations rather than as a fallback, and the trap is set
using some common utilities (ones not built into the user's shell).
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-01-26, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

you don't want to miss. - So Bash users, if they don't want to get
bitten in a subtle way, are probably advised to be pointed out to
that behavior, or, as suggested by others already, not to use tilde
at all with PATH in Bash.

That's a bizarre recommendation. Approximately all uses of tilde
in PATH are variants of PATH=~/bin:$PATH, which are expanded on
assignment in bash as in other sh-type shells.
--
Christian "naddy" Weisgerber naddy@mips.inka.de
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 26.01.2025 06:26, Kaz Kylheku wrote:

[...]

If someone has, say, "~/bin" in their PATH, ahead of /bin and /usr/bin,
I can put a malicious program in some directory called "~/bin"
somewhere in the filesystem, give that program the name of a common
external utility, and trick the user into changing into that location
where they will run this common command, resolving to my malicious
program.

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

Correct, but ...

So there would, IMO, not be a security hole (i.e. not because of that).

It's not a directory named '~/bin'. It's a directory named 'bin'
under a directory named '~'.

Bash interprets '~/bin' as a component of $PATH as $HOME/bin .
Everything(?) else interprets it as a relative path referring to
a bin subdirectory of a literal '~' subdirectory in the current
directory.

Hmm. The exploit Kaz discussed involves programs other than
bash treating '~/bin' as a relative path. But bash itself could
be affected if $HOME expands to a relative path (I've confirmed
the behavior). On the other hand, that's less likely to happen.
Kaz's exploit just requires getting the victim to cd into a specified directory; this would also require getting the user to change the
value of $HOME.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use rCLreorCY in a file/directory name.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vn6ifi$8je0$3@dont-email.me>,
Lawrence D'Oliveiro <ldo@nz.invalid> wrote:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use in a file/directory name.

That's a relief.
--
Kenny, I'll ask you to stop using quotes of mine as taglines.

- Rick C Hodgin -

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use rCLreorCY in a file/directory name.

Not in a POSIX-conforming way:

3.146 Filename
A sequence of bytes consisting of 1 to {NAME_MAX} bytes used to name a
file. The bytes composing the name shall not contain the <NUL> or
<slash> characters.

- https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_146


Alexis.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <87msfc7tgq.fsf@gmail.com>, Alexis <flexibeast@gmail.com> wrote: >Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use <line noise> in a file/directory name.

Not in a POSIX-conforming way:

Do you know what LDO meant by the above-quoted line noise?
--
I love the poorly educated.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On Mon, 27 Jan 2025 15:48:21 +1100, Alexis wrote:

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

But you can use rCLreorCY in a file/directory name.

Not in a POSIX-conforming way:

ldo@theon:trydir> mkdir f1
ldo@theon:trydir> touch f1/f2
ldo@theon:trydir> touch f1reof2
ldo@theon:trydir> ls -lR
.:
total 4
drwxr-xr-x 2 ldo users 4096 Jan 27 19:04 f1
-rw-r--r-- 1 ldo users 0 Jan 27 19:04 f1reof2

./f1:
total 0
-rw-r--r-- 1 ldo users 0 Jan 27 19:04 f2
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Mon, 27 Jan 2025 15:48:21 +1100, Alexis wrote:

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

But you can use rCLreorCY in a file/directory name.

Not in a POSIX-conforming way:

ldo@theon:trydir> mkdir f1
ldo@theon:trydir> touch f1/f2
ldo@theon:trydir> touch f1reof2

[...]

Yes, yes, we all know what you're saying, and we all hope you've
enjoyed the attention.

Of course the '/' character can appear in a pathname. And of course
it cannot appear in a pathname component, which POSIX also calls a
"filename".

pathname definition : https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_254

filename definition : https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_146

(On some systems it might be possible to edit the raw disk to
change a filename like "foo_bar" to "foo/bar". The result would
be a corrupted filesystem.)
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

gazelle@shell.xmission.com (Kenny McCormack) writes:

In article <87msfc7tgq.fsf@gmail.com>, Alexis <flexibeast@gmail.com> wrote:

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

But you can use <line noise> in a file/directory name.

Not in a POSIX-conforming way:

Do you know what LDO meant by the above-quoted line noise?

That, as Keith noted elsethread, they can appear in a POSIX 'pathname':

https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_254

My experience is that "file name" and "directory name" are typically
taken to refer to not the pathname / full path, but the output of applying basename(1) to that path. i was hoping to bring some precision to the discussion, but maybe i shouldn't have bothered.


Alexis.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <87ikq2bbf4.fsf@nosuchdomain.example.com>,
Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote: >cross@spitfire.i.gajendra.net (Dan Cross) writes:

In article <vn2hsj$2pe96$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

[snip]

Most of this discussion seems to be talking at cross-purposes.
I don't see much point in responding to the specifics.

My point was simply to show that shell behavior varies with
respect to how $PATH is treated; caveat emptor.

I don't usually use bash, but if you do and you really don't
like this behavior you can turn it off.

As far as I can tell, you can only turn off the behavior by
running in POSIX mode, which disables a lot of other Bash-specific >functionality. Personally, I'm unwilling to do that. I think I'd
like to see a "set -o" setting that disables just this feature.

That would be better, for sure.

You can *avoid* it by being careful not to put literal '~' characters
in $PATH (specifically at the beginning of any element of $PATH).
That's what I do (unintentionally before now, deliberately now that
I know about it.)

Indeed! I find it easiest just to not use `~` in $PATH. :-)

- Dan C.

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 26.01.2025 20:01, Christian Weisgerber wrote:

On 2025-01-26, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

you don't want to miss. - So Bash users, if they don't want to get
bitten in a subtle way, are probably advised to be pointed out to
that behavior, or, as suggested by others already, not to use tilde
at all with PATH in Bash.

That's a bizarre recommendation. Approximately all uses of tilde
in PATH are variants of PATH=~/bin:$PATH, which are expanded on
assignment in bash as in other sh-type shells.

Yes, to both. (Only that "recommendation" should have been put in
quotes; as it should have got obvious what I think about that.)

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 26.01.2025 18:51, Kaz Kylheku wrote:

On 2025-01-26, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.
So there would, IMO, not be a security hole (i.e. not because of that).

The / is not part of a name; it is acting as the path component
separator. [...]

Oh, I took your wording (emphasis by me)

*directory* called "~/bin"

literally as being meant the name of the directory (as opposed to
the complete path to the bin directory). - Thanks for clarifying.

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 27.01.2025 01:02, Lawrence D'Oliveiro wrote:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use rCLreorCY in a file/directory name.

You can use it in file and directory _paths_ as separator.

But how would you create (or subsequently access) any file
where '/' is part of its actual _name_?

Janis

--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-01-27, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use rCLreorCY in a file/directory name.

Yes, you can use UTF-8 encoded characters that look like ASCII in
a directory name, such as U+201C (left double quote), U+2215 (division
slash) and U+201D (right double quote).

Try not to interrupt when grownups are talking, though.
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-01-27, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Mon, 27 Jan 2025 15:48:21 +1100, Alexis wrote:

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

But you can use rCLreorCY in a file/directory name.

Not in a POSIX-conforming way:

ldo@theon:trydir> mkdir f1
ldo@theon:trydir> touch f1/f2
ldo@theon:trydir> touch f1reof2

[...]

Yes, yes, we all know what you're saying, and we all hope you've
enjoyed the attention.

He's making some irrelevant point about being able to use the
U+2215 "DIVISION SLASH" character in a path name, UTF-8 encoded.

Sure, Unicode character confusion can have security implications.

For instance, I'm guessing that if you call GetCommandLineA
on Windows on an input string containing U+2215, it will likely
map it to the ASCII slash. Which means that there is now a path
separator which wasn't there.

If a user is operating some program which calls another program, passing arguments to it, and that calling program validates against the use of
certain paths for security reasons, that user may be able to use this to
escape from the sandbox. For instance something like "../bar"
can be written with U+2215 slashes, so that it validates as a simple
file name; but upon mapping to ASCII, it becomes two components, one
of which escapes upward out of the working directory.

I don't suspect this problem intersects with the issue we are talking
about, but it's hard to be sure about a negative without doing a bunch
of work.

The ability to use a Unicode slash in a filename on POSIX systems,
thanks to UTF-8, is mostly a good thing, in my view.
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On Sun, 26 Jan 2025 23:51:33 -0800, Keith Thompson wrote:

Of course the '/' character can appear in a pathname. And of course it cannot appear in a pathname component, which POSIX also calls a
"filename".

But you can use rCLreorCY in a file/directory name.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On Mon, 27 Jan 2025 17:11:27 +0100, Janis Papanagnou wrote:

On 27.01.2025 01:02, Lawrence D'Oliveiro wrote:

On Sun, 26 Jan 2025 14:49:16 +0100, Janis Papanagnou wrote:

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

But you can use rCLreorCY in a file/directory name.

You can use it in file and directory _paths_ as separator.

No, the separator is the rCL/rCY character.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Sun, 26 Jan 2025 23:51:33 -0800, Keith Thompson wrote:

Of course the '/' character can appear in a pathname. And of course it
cannot appear in a pathname component, which POSIX also calls a
"filename".

But you can use rCLreorCY in a file/directory name.

Yes, you've already told that joke.

Again, that's a DIVISION SLASH character, U+2215, not a SOLIDUS or SLASH character, U+002F. It's the latter that cannot appear in a filename.

You probably think you're being clever.

As Kaz recently told you, "Try not to interrupt when grownups are talking".
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

In article <vn93nt$19o0q$7@dont-email.me>,
Lawrence D'Oliveiro <ldo@nz.invalid> wrote:

On Sun, 26 Jan 2025 23:51:33 -0800, Keith Thompson wrote:

Of course the '/' character can appear in a pathname. And of course it
cannot appear in a pathname component, which POSIX also calls a
"filename".

But you can use in a file/directory name.

But you can use r^@^\r^H^Ur^@^] (line noise) in a file/directory name.
--
Politics is show business for ugly people.

Sports is politics for stupid people.
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

On 26.01.2025 06:26, Kaz Kylheku wrote:

[...]

If someone has, say, "~/bin" in their PATH, ahead of /bin and /usr/bin,
I can put a malicious program in some directory called "~/bin"
somewhere in the filesystem, give that program the name of a common
external utility, and trick the user into changing into that location
where they will run this common command, resolving to my malicious
program.

To my best knowledge using '/' as part of a file or directory name is
(as the '\0') prohibited by the operating system at a very low level.

Correct, but ...

So there would, IMO, not be a security hole (i.e. not because of that).

It's not a directory named '~/bin'. It's a directory named 'bin'
under a directory named '~'.

Bash interprets '~/bin' as a component of $PATH as $HOME/bin .
Everything(?) else interprets it as a relative path referring to
a bin subdirectory of a literal '~' subdirectory in the current
directory.

Hmm. The exploit Kaz discussed involves programs other than
bash treating '~/bin' as a relative path. But bash itself could
be affected if $HOME expands to a relative path (I've confirmed
the behavior). On the other hand, that's less likely to happen.
Kaz's exploit just requires getting the victim to cd into a specified directory; this would also require getting the user to change the
value of $HOME.

Another interesting tidbit: the GNU `which` command has a "--skip-tilde"
option that tells it to skip elements of $PATH that start with '~'
(and also directories anywhere under $HOME).

From the output of `which --help`:

--skip-tilde Skip directories in PATH that start with a tilde.

From the info documentation:

rCy--skip-tilderCO
Skip directories in rCyPATHrCO that start with a tilde and executables
which reside in the rCyHOMErCO directory.

(On Ubuntu, /usr/bin/which is provided by the the "debianutils" package.
I don't see a way to install GNU which other than by building it from
source.)
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: comp.unix.shell

On 2025-02-03, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:

Another interesting tidbit: the GNU `which` command has a "--skip-tilde" option that tells it to skip elements of $PATH that start with '~'
(and also directories anywhere under $HOME).

From the output of `which --help`:

--skip-tilde Skip directories in PATH that start with a tilde.

Le's get this straight. The programmer who wrote this (we need not
repeat his name again, since it came up here about twice recently)
noticed that he had unexpanded tildes leaking into his PATH environment variable, visible to child processes of his shell. Yet, instead of
realizing how wong that is and possibly reporting a bug against Bash,
he just ... hacked a feature into his pet utility to ignore them.

Everything to do with this utility is just whichcraft as far as I'm
concerned. (Okay, that is worn threadbare now, I will stop).
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
--- Synchronet 3.21b-Linux NewsLink 1.2