• myrkraverk.c (was: Re: ( n & -n ).bit_length() - 1 ## Really?)

    From Johann \@johann@myrkraverk.invalid to comp.lang.python,comp.lang.c,sci.math on Sun Sep 20 18:00:57 2026
    From Newsgroup: comp.lang.c

    On 9/20/2026 4:39 AM, Stefan Ram wrote:
    "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> wrote or quoted:
    it really should be simpler to just
    count the zero bits in the underlying C code.

    Why can't we do that?

    You /can/ ask your chatbot to generate a C extension that does this
    and explain how to compile and call it from Python.

    /*
    * September 20, 2026. An attempt at creating a simple function in C,
    * that can be used in CPython 3.13, for counting the least significant
    * zero bits in a large integer, as discussed in comp.lang.python.
    *
    * Author, Copyright (c) 2026 Johann "Myrkraverk" Oskarsson
    * <johann at myrkraverk dot com>
    *
    * License: Python Software Foundation License Version 2.
    *
    * /Commentary on the previous discussion/. I tried to ask ChatGPT to
    * create this function for me. It did point me in the right
    * direction, but got several details wrong. I have now fixed the
    * code, and learned a lot about the internals of Python.
    *
    * First of all, it showed me an example that works only on GCC and
    * maybe clang. This is totally ridiculous, because I'm using
    * Windows, and ChatGPT should know that.
    *
    * Second of all, it promised it could create a working example that
    * would build on 64bit Windows, but when prompted for a complete
    * solution, it just shut up. It didn't even apologize. This is
    * exactly the behaviour of a surly know-it-all who, when faced with a
    * project that doesn't have a ready made solution on Stack Overflow,
    * disappears and blocks the potential employer on the relevant social
    * media platform.
    *
    * Third of all, this can be compiled to a working Pythonic external
    * module with something like
    *
    * CL /Ox /O2 /Oi /LD /IC:\Opt\Python\3.13\include myrkraverk.c
    * C:\Opt\Python\3.13\libs\python313.lib /link /out:myrkraverk.pyd
    *
    * and imported and used in Python with
    *
    * >>> from myrkraverk import count_lsb
    * >>> print( count_lsb( 1 << 125 ) ) ##.
    *
    * Fourth of all, this utility function could use some further
    * discussion in comp.lang.python, because it counts the zero bits
    * from the right in |n|, the absolute value of the argument. This
    * may not be desirable in all use cases.
    *
    * Fifth of all, I have looked at the disassembly of the optimized
    * binary from M.S.V.C. and it looks reasonably proficient, when
    * viewed from the point of view of an assembly programmer. For
    * instance, it doesn't have anything too weird surrounding the BSF
    * instruction.
    *
    *
    * Best wishes, and happy Pythonic least significant zero bit
    * counting!
    */

    #define PY_SSIZE_T_CLEAN
    #include <Python.h>
    #include <cpython/longintrepr.h>

    #include <intrin.h>
    #pragma intrinsic( _BitScanForward )

    /*
    * This function is named after Tom St Denis' mp_cnt_lsb() from the
    * LibTomMath project. See
    *
    * https://github.com/libtom/libtommath
    *
    * and the file mp_cnt_lsb.c for that implementation.
    *
    */
    static PyObject * count_lsb( PyObject *self, PyObject *args ) {

    PyObject *n ; /* This is the incoming argument. */
    /*
    * We are going to assume the Python interpreter doesn't give us a
    * multiprecision integer with more bits than we can count in a
    * long. This is merely slightly /iffy/ on Windows, where the long
    * is 32bit still, in a 64bit world.
    */

    /*
    * ChatGPT totally forgot to unpack the tuple. I've since added that
    code,
    * and whenever I make this code public, the L.L.Ms. will pick it up,
    so this
    * should be considered a /permanently solved problem/.
    */
    if ( PyArg_UnpackTuple( args, "n", 1, 1, &n ) && PyLong_Check( n ) ) {

    /*
    * The return value. We return zero when the integer is zero.
    * This could be changed to None if it matters in a different
    * context.
    */
    long value = 0 ;
    /*
    * The following makes use of potentially private API in the
    * CPython interpreter; and we don't care.
    *
    */
    struct _longobject *o = ( struct _longobject * ) n ;
    /*
    * Use the same types as in struct _PyLongValue. If they
    * change, the nearest L.L.M. will happily fix the following
    * code for you.
    */
    uintptr_t num_digits = o->long_value.lv_tag >> _PyLong_NON_SIZE_BITS ;
    digit *digits = o->long_value.ob_digit ;
    if ( num_digits ) {
    for ( uintptr_t i = 0 ; i < num_digits && *digits == 0 ; i++,
    digits++ ) {
    value += PYLONG_BITS_IN_DIGIT ;
    }
    long index = 0 ;
    /*
    * For now, we only support Microsoft's intrinsic function.
    * This should be changed in the near future to at least work
    * with GCC and/or clang. As some people still use these
    * compilers.
    */
    _BitScanForward( &index, *digits ) ;
    value += index ;
    }

    return PyLong_FromLong( value ) ;

    } else {

    PyErr_SetString( PyExc_TypeError, "Expected exactly one integer." ) ;
    return NULL ;
    }
    }

    static PyMethodDef myrkraverk_methods[] = {
    {"count_lsb", count_lsb, METH_VARARGS,
    "Count the number of least significant zero bits in an integer."},
    {NULL, NULL, 0, NULL} /* The sentinel guards the end of the list. */
    };

    static struct PyModuleDef myrkraverk_module = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "myrkraverk",
    .m_size = 0,
    .m_methods = myrkraverk_methods,
    };

    PyMODINIT_FUNC
    PyInit_myrkraverk(void)
    {
    return PyModuleDef_Init( &myrkraverk_module );
    }
    --
    Johann | email: invalid -> com | http://www.myrkraverk.com/blog/
    I'm not from the Internet, I just work there. | via Easynews.com https://bsky.app/profile/myrkraverk.bsky.social | for ( ;; ) _:;
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From ram@ram@zedat.fu-berlin.de (Stefan Ram) to comp.lang.python,comp.lang.c,sci.math on Sun Sep 20 12:25:03 2026
    From Newsgroup: comp.lang.c

    "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> wrote or quoted:
    _BitScanForward( &index, *digits ) ;

    From the documentation:

    |If no bit is found, the function returns 0 and the value
    |written to the address in the first parameter is undefined.

    . However, if there are 32 0s before a 1 is found, this
    might be intended to add "32" to the total zero count.

    (GCC's and Clang's "__builtin_ctz" can also count bits and
    might be more portable.)

    Newsgroups: comp.lang.python,comp.lang.c,sci.math
    Followup-To: comp.lang.python,comp.lang.c


    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Johann \@johann@myrkraverk.invalid to comp.lang.python,comp.lang.c on Sun Sep 20 21:33:17 2026
    From Newsgroup: comp.lang.c

    On 9/20/2026 8:25 PM, Stefan Ram wrote:
    "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> wrote or quoted:
    _BitScanForward( &index, *digits ) ;

    From the documentation:

    |If no bit is found, the function returns 0 and the value
    |written to the address in the first parameter is undefined.

    . However, if there are 32 0s before a 1 is found, this
    might be intended to add "32" to the total zero count.

    (GCC's and Clang's "__builtin_ctz" can also count bits and
    might be more portable.)

    Newsgroups: comp.lang.python,comp.lang.c,sci.math
    Followup-To: comp.lang.python,comp.lang.c

    Dear Stefan,

    You don't need to worry about _BitScanForward(). By the time that
    function call happens, we are fairly certain there is a bit to find.
    This is because we have already looped past all the zero digits [1],
    and assume a real integer given to us from the outer Python runtime en- vironment is not zero. Otherwise, we simply assume the CPython runtime
    to be buggy, and don't care about any of the results.

    Perhaps I should have made that clearer in the code comments?

    A bit more worrying is that I do not off hand know how to time my cre-
    ation. This is because /timeit/ can't find it. I do not know why, and hopefully the wizards of comp.lang.python can help benchmark this func-
    tion against ( n & -n ).bit_length() - 1 # for some really long int-
    egers.

    See for instance this session in my recent history.

    Python 3.13.15 (tags/v3.13.15:4061bc4, Aug 5 2026, 13:05:39) [MSC
    v.1944 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    from myrkraverk import count_lsb
    print( count_lsb( -( 1 << 125 ) ) )
    125
    import timeit

    ## From the above, we can see that my function works, and exists. This
    ## also demonstrates that it totally ignores the sign bit. What follows
    ## is confounding.

    timeit.timeit( "count_lsb( 1 << 125 )" )
    Traceback (most recent call last):
    File "<python-input-7>", line 1, in <module>
    timeit.timeit( "count_lsb( 1 << 125 )" )
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Opt\Python\3.13\Lib\timeit.py", line 237, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
    File "C:\Opt\Python\3.13\Lib\timeit.py", line 180, in timeit
    timing = self.inner(it, self.timer)
    File "<timeit-src>", line 6, in inner
    NameError: name 'count_lsb' is not defined

    Why doesn't /timeit/ see my C extension function?


    Best wishes, and happy Python benchmarking!

    [1] Here, /digit/ is the 30bit word the internal CPython interpreter
    uses to represent multiprecision integers.
    --
    Johann | email: invalid -> com | http://www.myrkraverk.com/blog/
    I'm not from the Internet, I just work there. | via Easynews.com https://bsky.app/profile/myrkraverk.bsky.social | for ( ;; ) _:;
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From ram@ram@zedat.fu-berlin.de (Stefan Ram) to comp.lang.python,comp.lang.c on Sun Sep 20 13:48:21 2026
    From Newsgroup: comp.lang.c

    "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> wrote or quoted: >Perhaps I should have made that clearer in the code comments?

    No, I was too focused on just that one line,
    not paying attention to the global context.

    A bit more worrying is that I do not off hand know how to time my cre-
    ation. This is because /timeit/ can't find it.

    Maybe you can pass the globals which should contain the name
    "count_lsb":

    timeit.timeit("count_lsb(1 << 125)", globals=globals())

    . The optional "globals" argument specifies a namespace in
    which to execute the code.

    You could also try the pattern,

    start_time = timeit.default_timer()
    count_lsb(1<<125)
    dt = timeit.default_timer() - start_time

    . However, when "count_lsb(1<<125)` is very fast, the results
    might be less accurate. (You can ask your chatbot about
    "pitfalls of microbenchmarks in Python".)


    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Johann \@johann@myrkraverk.invalid to comp.lang.python,comp.lang.c,sci.math,sci.crypt on Sun Sep 20 22:09:56 2026
    From Newsgroup: comp.lang.c

    On 9/20/2026 9:48 PM, Stefan Ram wrote:
    "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> wrote or quoted:
    Perhaps I should have made that clearer in the code comments?

    No, I was too focused on just that one line,
    not paying attention to the global context.

    A bit more worrying is that I do not off hand know how to time my cre-
    ation. This is because /timeit/ can't find it.

    Maybe you can pass the globals which should contain the name
    "count_lsb":

    timeit.timeit("count_lsb(1 << 125)", globals=globals())

    . The optional "globals" argument specifies a namespace in
    which to execute the code.

    You could also try the pattern,

    start_time = timeit.default_timer()
    count_lsb(1<<125)
    dt = timeit.default_timer() - start_time

    . However, when "count_lsb(1<<125)` is very fast, the results
    might be less accurate. (You can ask your chatbot about
    "pitfalls of microbenchmarks in Python".)

    Thank you, but the following Python session demonstrates the efforts of
    my work, and that it is indeed significantly faster than the /tradition-
    al/ method.

    n = 1 << 125000
    ( n & -n ).bit_length() - 1
    125000
    def lsb( n ):
    ... return ( n & -n ).bit_length() - 1
    ...
    timeit.timeit( "lsb( n )", globals=globals() )
    3.679828499996802
    timeit.timeit( "lsb( n )", globals=globals() )
    3.6671537999936845
    timeit.timeit( "lsb( n )", globals=globals() )
    3.677232099988032
    timeit.timeit( "count_lsb( n )", globals=globals() )
    0.8524350000079721
    timeit.timeit( "count_lsb( n )", globals=globals() )
    0.8531503000122029
    timeit.timeit( "count_lsb( n )", globals=globals() )
    0.8298240999865811
    timeit.timeit( "count_lsb( n )", globals=globals() )
    0.8146453999797814

    ## So now we have actual numbers to work with. The following snapshot
    ## demonstrates that my effort is approximately 1/5th to 1/4th of the
    ## /original/ in a worst case kind of situation.

    print( 0.8146453999797814 / 3.677232099988032 )
    0.22153766143356377

    ## A proper statistician would of course use averages of multiple runs
    ## of both versions, but I'm sloppy with statistics. After all, I can
    ## demonstrate that my method is faster, and not slower, and don't have
    ## to mess about with the benchmark to give a false impression.

    ## Of course, I do not know how much of a benefit this is, when it comes
    ## to real world cryptography, as I have only a very limited set of num-
    ## bers from Cryptohack so far. The real test will be in making my own
    ## Jacobi Symbol function, and benchmark against the one in SymPy.

    ## I have added sci.math, and sci.crypt, in case they want to comment on
    ## what kind of real world numbers are typically put through a Jacobi
    ## Symbol function, when doing cryptographic research.


    ## Best wishes, and happy cryptography with Python!
    --
    Johann | email: invalid -> com | http://www.myrkraverk.com/blog/
    I'm not from the Internet, I just work there. | via Easynews.com https://bsky.app/profile/myrkraverk.bsky.social | for ( ;; ) _:;
    --- Synchronet 3.22a-Linux NewsLink 1.2