c_uint32 bitfields break structures

[RonyBatista]

BPO	24290
Nosy	@ronaldoussoren, @ned-deily
Files	ctypesBug.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-05-26.19:02:48.705>
labels = ['ctypes', 'type-bug']
title = 'c_uint32 bitfields break structures'
updated_at = <Date 2015-05-27.22:57:25.193>
user = 'https://bugs.python.org/RonyBatista'

bugs.python.org fields:

activity = <Date 2015-05-27.22:57:25.193>
actor = 'Rony Batista'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2015-05-26.19:02:48.705>
creator = 'Rony Batista'
dependencies = []
files = ['39508']
hgrepos = []
issue_num = 24290
keywords = []
message_count = 5.0
messages = ['244125', '244126', '244127', '244132', '244223']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'Rony Batista']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24290'
versions = ['Python 2.7']

[RonyBatista]

ctypes Structures with c_uint32 bitfields have strange behaviour on OS X.

In the attached code when the field "number" is set, it changes the whole 32 bits, even thou its meant to be 23 bits.

The behavior is unexpected in:
OS X: Python 2.7.9

The behavior is as expected in the following environments:
Windows: python 2.7.9
Ubuntu: python 2.7.6

[ronaldoussoren]

You forgot to actually attach the code.

[RonyBatista]

Silly me, Heres the code.

[ned-deily]

Without diving into the details of your test program, I get the same results on a 64-bit Debian Python 2.7.9 as with a 64-bit OS X 2.7.10:

c_uint32 TESTS:
Class Name exponent number Sign float binary repr
IEEE754Float_u ('-0x7b', '0xbeb2aL', 0L, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2aL', 0L, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('0x0', '0xbeb2a', 0L, 1.094551427887186e-39, ['0x0', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('0x0', '0xbeb2a', 0L, 1.094551427887186e-39, ['0x0', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0L, 1.094551427887186e-39, ['0x85', '0xb', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2aL', 1L, -69.95930480957031, ['0xc2', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 1L, 1.094551427887186e-39, ['0x85', '0x8b', '0xeb', '0x2a'])

But using the same OS X 2.7.10 in 32-bit mode, I get:

c_uint32 TESTS:
Class Name exponent number Sign float binary repr
IEEE754Float_u ('-0x7b', '0xbeb2aL', 0L, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2aL', 0L, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0L, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0L, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0L, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2aL', 1L, -69.95930480957031, ['0xc2', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 1L, -69.95930480957031, ['0x85', '0x8b', '0xeb', '0x2a'])

[RonyBatista]

Well, looks like the issue is with 64 bit mode then.

For the first 5 cases the right answer is 69.95930480957031, and for the last 2 its -69.95930480957031. The results for the 32 bit mode are all correct.

[encukou]

Should be fixed in #97702.

Code updated for Python 3:

__author__ = 'rony'

from ctypes import BigEndianStructure
from ctypes import sizeof, memmove, addressof
from ctypes import c_int32, c_uint32
from struct import pack, unpack


class BigEndianByteStructure(BigEndianStructure):
    # https://wiki.python.org/moin/ctypes
    @classmethod
    def from_bytes(cls, ibytes):
        return_val = cls()
        if len(ibytes) != sizeof(return_val):
            raise ValueError(
                '{0} Cannot unpack {2} bytes, it should be {1} bytes'.format(cls.__name__, sizeof(return_val),
                                                                             len(ibytes)))

        memmove(addressof(return_val), ibytes, len(ibytes))
        return return_val

    @classmethod
    def from_structure(cls, original):
        return_val = cls()
        for field in return_val._fields_:
            try:
                setattr(return_val, field[0], getattr(original, field[0]))
            except:
                print( 'Attribute {0} not found in original'.format(field[0]))

        return return_val


# POSITIVE
# IEEE754 HEX    0x428beb2a
# AN575 HEX      0x850beb2a

# NEGATIVE
# IEEE754 HEX    0xc28beb2a
# AN575 HEX      0x858beb2a

class IEEE754Float(BigEndianByteStructure):
    _pack_ = 1
    _fields_ = [('sign', c_int32, 1),
                ('exponent', c_int32, 8),
                ('number', c_int32, 23)]

    @classmethod
    def from_float(cls, f):
        return cls.from_bytes(pack('>f', f))

    def __float__(self):
        return unpack('>f', memoryview(self))[0]

    def debug(self):
        f = self
        return hex(f.exponent), hex(f.number), f.sign, float(f), [hex(b) for b in bytearray(f)]


class AN575Float(BigEndianByteStructure):
    _pack_ = 1
    _fields_ = [('exponent', c_int32, 8),
                ('sign', c_int32, 1),
                ('number', c_int32, 23)]

    @classmethod
    def from_float(cls, f):
        return cls.from_structure(IEEE754Float.from_float(f))

    def __float__(self):
        a = IEEE754Float.from_structure(self)
        return float(a)

    def debug(self):
        f = self
        return hex(f.exponent), hex(f.number), f.sign, float(f), [hex(b) for b in bytearray(f)]


class IEEE754Float_u(BigEndianByteStructure):
    _pack_ = 1
    _fields_ = [('sign', c_uint32, 1),
                ('exponent', c_int32, 8),
                ('number', c_uint32, 23)]

    @classmethod
    def from_float(cls, f):
        return cls.from_bytes(pack('>f', f))

    def __float__(self):
        return unpack('>f', memoryview(self))[0]

    def debug(self):
        f = self
        return hex(f.exponent), hex(f.number), f.sign, float(f), [hex(b) for b in bytearray(f)]


class AN575Float_uint(BigEndianByteStructure):
    _pack_ = 1
    _fields_ = [('exponent', c_int32, 8),
                ('sign', c_uint32, 1),
                ('number', c_int32, 23)]

    @classmethod
    def from_float(cls, f):
        return cls.from_structure(IEEE754Float_u.from_float(f))

    def __float__(self):
        a = IEEE754Float_u.from_structure(self)
        return float(a)

    def debug(self):
        f = self
        return hex(f.exponent), hex(f.number), f.sign, float(f), [hex(b) for b in bytearray(f)]


print ('\n', 'c_int32 TESTS:')
print ('Class Name\t\t', 'exponent', '\tnumber', '\tSign', '\tfloat', '\t\t', 'binary repr')
f = IEEE754Float.from_float(69.9593067)
print (type(f).__name__, f.debug())

f = IEEE754Float.from_bytes(b'\x42\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

f = AN575Float.from_float(69.9593067)
print (type(f).__name__, f.debug())

f = AN575Float.from_structure(IEEE754Float.from_float(69.9593067))
print (type(f).__name__, f.debug())

f = AN575Float.from_bytes(b'\x85\x0b\xeb\x2a')
print (type(f).__name__, f.debug())

f = IEEE754Float.from_bytes(b'\xc2\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

f = AN575Float.from_bytes(b'\x85\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

print ('\n', 'c_uint32 TESTS:')
print ('Class Name\t\t', 'exponent', '\tnumber', '\tSign', '\tfloat', '\t\t', 'binary repr')
f = IEEE754Float_u.from_float(69.9593067)
print (type(f).__name__, f.debug())

f = IEEE754Float_u.from_bytes(b'\x42\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

f = AN575Float_uint.from_float(69.9593067)
print (type(f).__name__, f.debug())

f = AN575Float_uint.from_structure(IEEE754Float.from_float(69.9593067))
print (type(f).__name__, f.debug())

f = AN575Float_uint.from_bytes(b'\x85\x0b\xeb\x2a')
print (type(f).__name__, f.debug())

f = IEEE754Float_u.from_bytes(b'\xc2\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

f = AN575Float_uint.from_bytes(b'\x85\x8b\xeb\x2a')
print (type(f).__name__, f.debug())

Result (same on 64- and 32-bit OS):

 c_int32 TESTS:
Class Name		 exponent 	number 	Sign 	float 				 binary repr
IEEE754Float ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
IEEE754Float ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
AN575Float ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
IEEE754Float ('-0x7b', '0xbeb2a', -1, -69.95930480957031, ['0xc2', '0x8b', '0xeb', '0x2a'])
AN575Float ('-0x7b', '0xbeb2a', -1, -69.95930480957031, ['0x85', '0x8b', '0xeb', '0x2a'])

 c_uint32 TESTS:
Class Name		 exponent 	number 	Sign 	float 				 binary repr
IEEE754Float_u ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x42', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 0, 69.95930480957031, ['0x85', '0xb', '0xeb', '0x2a'])
IEEE754Float_u ('-0x7b', '0xbeb2a', 1, -69.95930480957031, ['0xc2', '0x8b', '0xeb', '0x2a'])
AN575Float_uint ('-0x7b', '0xbeb2a', 1, -69.95930480957031, ['0x85', '0x8b', '0xeb', '0x2a'])