Myself I'm usually not using CSV format(s), but recently I advertised
GNU Awk (given that newer versions support CSV data processing) to a
friend seeking CSV solutions.

I was quite astonished when I stumbled across a StackOverflow article
about CSV processing with contemporary versions of GNU Awk and read
that you are restricted to comma as separator and double quotes to
enclose strings. The workarounds provided at SO were extremely clumsy.

Given that using ',', ';', '|' (or other delimiters) and also various
types of quotes are just a lexical (no functional) difference I wonder
whether it would be sensible to be able to define them, say, through
setting a PROCINFO element?

Janis

https://stackoverflow.com/questions/45420535/whats-the-most-robust-way-to-efficiently-parse-csv-using-awk

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 26.08.2024 13:26, Ed Morton wrote:

On 8/25/2024 1:00 AM, Janis Papanagnou wrote:

Myself I'm usually not using CSV format(s), but recently I advertised
GNU Awk (given that newer versions support CSV data processing) to a
friend seeking CSV solutions.

I was quite astonished when I stumbled across a StackOverflow article
about CSV processing with contemporary versions of GNU Awk and read
that you are restricted to comma as separator and double quotes to
enclose strings. The workarounds provided at SO were extremely clumsy.

Given that using ',', ';', '|' (or other delimiters) and also various
types of quotes are just a lexical (no functional) difference I wonder
whether it would be sensible to be able to define them, say, through
setting a PROCINFO element?

Janis

https://stackoverflow.com/questions/45420535/whats-the-most-robust-way-to-efficiently-parse-csv-using-awk

FYI gawk just inherited those behaviors (plus mandatory stripping of the quotes from quoted fields, see https://lists.gnu.org/archive/html/bug-gawk/2023-11/msg00018.html) from Kernighans awk.

Thanks.

My opinion on this is that I wouldn't expect GNU Awk to become a (yet
another) CSV-processor. It's very convenient to have an easy input of
CSV data to be processed like other tabular data with Awk. So removal
of the (outer) quotes, transforming "inner" quotes of fields according
to the CSV-standard(s), and handling the escape symbol, would serve my expectations. (I don't need CSV-output formatting, but I understand if
there is such a demand.)

The (flexible) support for (at least typical) field separators is IMO
more pressing. Whether it can be supported by PROCINFO[] (as I've [ad
hoc] written above) or using FS is only a detail.

But given the current implementation, this error message

$ awk -F';' --csv '...' data.csv
awk: warning: assignment to FS/FIELDWIDTHS/FPAT has no effect when using
--csv

indicates that there's already some FS consistency logic existing, so
instead of introducing another PROCINFO attribute it would probably be
more obvious and appear more consistent to the user to use ("re-use")
FS for the purpose of defining the CSV field delimiter.

And we also don't need to explicitly PROCINFO-define the used quotes
since the quotes could anyway be identified (and handled) implicitly.
Or are there any issues with data like

"Hi there!",42,'Hello "world"?',"Ed's post",3.14

to be provided (in $1..$5) as

Hi there!,42,Hello "world"?,Ed's post,3.14


Where I have a general unsettling feeling is when locales influence
the processing. Personally I have defined "LC_NUMERIC=C.UTF-8" so my
real numbers use a decimal point for the fraction anyway, but if the
processed data will have a locale numbers-representation with commas
then there might be issues (and not only with the CSV commas).

I'd have liked to provide more concrete information here, but I'm at
the moment even unable to reproduce Awk's behavior as documented in
its manual; I've tried the following command with various locales

$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
-| 5,321

but always got just 5 as result.

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 26.08.2024 19:01, Manuel Collado wrote:

El 26/8/24 a las 14:54, Janis Papanagnou escribió:

My opinion on this is that I wouldn't expect GNU Awk to become a (yet
another) CSV-processor. It's very convenient to have an easy input of
CSV data to be processed like other tabular data with Awk. So removal
of the (outer) quotes, transforming "inner" quotes of fields according
to the CSV-standard(s), and handling the escape symbol, would serve my
expectations. (I don't need CSV-output formatting, but I understand if
there is such a demand.)

Perhaps you could try my gawk-csvio pure gawk library. Just include a
first unconditional rule calling csvimport(...) and the CSV input data
will be automatically converted to a regular OFS-delimited record ready
to be processed.

Please find the library at http://mcollado.z15.es/gawk-extras/. The documentation is available also to be read before downloading.

As said upthread: "Myself I'm usually not using CSV format(s), [...]".
But thanks, I'll forward that information to interested people I know.

Just to clarify; my above stated opinion was meant to provide a view
concerning the newly incorporated '--csv' feature to obtain a greater acceptance and not disappointment from people choosing and using GNU
Awk for that purpose. Given how it got implemented it's (IMO) of very
limited use (if you cannot control the delimiters and quotes).

If I understand correctly that the library you mention would address
the two topics (field separator and quoting) then there's even less a
point (I suppose) to use the new '--csv' option in GNU Awk; just use
your library instead?

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 27.08.2024 02:49, Ed Morton wrote:

On 8/26/2024 7:54 AM, Janis Papanagnou wrote:

snip>
I'd have liked to provide more concrete information here, but I'm at
the moment even unable to reproduce Awk's behavior as documented in
its manual; I've tried the following command with various locales

$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
-| 5,321

but always got just 5 as result.

You need to specifically TELL gawk to use your locale to read input
numbers:

$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
5

$ echo 4,321 | POSIXLY_CORRECT=1 LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }' 5,321

$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk -N '{ print $1 + 1 }' 5,321

See https://www.gnu.org/software/gawk/manual/gawk.html#Locale-influences-conversions
for more info on that.

Thanks. That's actually where I got above example from.

I've missed that there was an explicit
$ export POSIXLY_CORRECT=1
set on the very top of these examples. Gee!

Feels anyway strange that an explicit LC_* setting is ineffective
without the additional POSIXLY_CORRECT variable. And the page also
says: "The POSIX standard says that awk always uses the period as
the decimal point when reading the awk program source code".
So despite POSIX saying that, you have to use a variable named
POSIXLY_CORRECT. - Do I need some more coffee to understand that?

And I see there's an additional GNU Awk option '--use-lc-numeric'.
What a mess!

(I suppose current status can only be explained by the mentioned
forth-and-back during history of various GNU Awk versions.)

What's worth the LC_* variables if they are ignored (or maybe not).

Janis

Regards,

Ed

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 27.08.2024 13:45, Ed Morton wrote:

On 8/26/2024 8:39 PM, Janis Papanagnou wrote:

I've missed that there was an explicit
$ export POSIXLY_CORRECT=1
set on the very top of these examples. Gee!

POSIXLY_CORRECT=1 (or equivalently `--posix` aka `-P`) affects numbers
in the input your script reads (as shown in the previous post) and
strings being converted to numbers in your code, it doesn't affect
literal numbers in the source code for your script that awk reads.

In the source code the decimal separator for a literal number (as
opposed to a string being converted to a number) is always `.`.

You can't use, say, a comma as the decimal separator in a literal number because a comma already means something in the awk syntax, e.g. `print
4,321` means the same as "print 4 OFS 321`.

[examples and explanations]

Thanks.

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 27.08.2024 12:20, Manuel Collado wrote:

El 27/8/24 a las 2:31, Janis Papanagnou escribió:

If I understand correctly that the library you mention would address
the two topics (field separator and quoting) then there's even less a
point (I suppose) to use the new '--csv' option in GNU Awk; just use
your library instead?

A user decision.

No doubt.

HTH

Not the least for someone trying to understand any advantage of '--csv'
given the existence of your (obviously more generalized) CSV library.

Janis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)