From Newsgroup: comp.lang.python

So in kate, i Ctrl-Shift-U and type ffff to enter a unicode codepoint of 0xffff. Then at the REPL prompt i do
fh = open('/tmp/x', 'rt', -1, 'ascii', 'backslashreplace', None)

and i get
fh.readline()
'\\xef\\xbf\\xbf\n'

Since I wrote two bytes 0xff and 0xff into Kate - why am i getting 0xef
0xbf and 0xbf ?
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.python

On Sat, 8 Aug 2026 00:38:24 -0000 (UTC), Veek M wrote:

So in kate, i Ctrl-Shift-U and type ffff to enter a unicode
codepoint of 0xffff. Then at the REPL prompt i do
fh = open('/tmp/x', 'rt', -1, 'ascii', 'backslashreplace', None)

and i get
fh.readline()
'\\xef\\xbf\\xbf\n'

Since I wrote two bytes 0xff and 0xff into Kate - why am i getting
0xef 0xbf and 0xbf ?

Python 3.14.6 (main, Jun 10 2026, 18:54:31) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'\xef\xbf\xbf\n'.decode()
'\uffff\n'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.python

On Sat, 8 Aug 2026 01:19:32 -0000 (UTC), Lawrence DrCOOliveiro wrote:

b'\xef\xbf\xbf\n'.decode()

Could you explain how it works and what exactly is going on?

fh.readline() returns a unicode string with the funny chars (bytes 0xff
0xff) encoded as \\xef \\xbf \\xbf - why is it \\? why not just use a
single u'\xef\xbf\xbf' - why is he escaping the '\'.

Also - how exactly is he getting ef bf bf and not ff ff?
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.python

On Sat, 8 Aug 2026 04:22:59 -0000 (UTC), Veek M wrote:

On Sat, 8 Aug 2026 01:19:32 -0000 (UTC), Lawrence DrCOOliveiro wrote:

b'\xef\xbf\xbf\n'.decode()

Could you explain how it works and what exactly is going on?

fh.readline() returns a unicode string with the funny chars (bytes 0xff
0xff) encoded as \\xef \\xbf \\xbf - why is it \\? why not just use a
single u'\xef\xbf\xbf' - why is he escaping the '\'.

Also - how exactly is he getting ef bf bf and not ff ff?

oh is 0xff 0xff when encoded to disk in utf-8
(sys.getsystemdefaultencoding) 0xef 0xbf 0xbf?
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.python

On Sat, 8 Aug 2026 04:25:43 -0000 (UTC), Veek M wrote:

On Sat, 8 Aug 2026 04:22:59 -0000 (UTC), Veek M wrote:

On Sat, 8 Aug 2026 01:19:32 -0000 (UTC), Lawrence DrCOOliveiro wrote:

b'\xef\xbf\xbf\n'.decode()

Could you explain how it works and what exactly is going on?

fh.readline() returns a unicode string with the funny chars (bytes 0xff
0xff) encoded as \\xef \\xbf \\xbf - why is it \\? why not just use a
single u'\xef\xbf\xbf' - why is he escaping the '\'.

Also - how exactly is he getting ef bf bf and not ff ff?

oh is 0xff 0xff when encoded to disk in utf-8
(sys.getsystemdefaultencoding) 0xef 0xbf 0xbf?

yes,
root@laptopveek:/tmp# od -x /tmp/x
0000000 bfef 0abf
0000004

it's the raw utf-8 encoded as bytes but since it is a unicode string why doesn't he save it as u'\xef\xbf\xbf' why does he escape the '\' and make
it '\\x'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.python

On 8/08/26 4:33 pm, Veek M wrote:

it's the raw utf-8 encoded as bytes but since it is a unicode string why doesn't he save it as u'\xef\xbf\xbf' why does he escape the '\' and make
it '\\x'

Because you decoded it as ascii with backslashreplace. It's replacing
each byte that's outside the ascii range with four characters: a
backslash, an 'x', and two hex digits. The backslashes are being doubled
when you print the string and its repr() gets computed.

Since the file is actually utf-8 and not ascii, that's the appropriate
way to decode it:

fh = open('/tmp/x', 'rt', encoding = 'utf-8')

Then your ffff should come through as a single character in the string
and print as '\uffff'.
--
Greg




--- Synchronet 3.22a-Linux NewsLink 1.2