Monday, November 05, 2012

OS X xmm/ymm registers in gdb

I made it to the floating point math chapter in my book and discovered that ebe was showing 0's for the xmm registers and wouldn't display the ymm registers.  I checked and a program which loaded 3.25 into xmm0 ended up with xmm0.v4_float = {0.0, 0.0, 0.0, 3.25} while the same program under Linux shows it as {3.25, 0.0, 0.0, 0.0}.  I don't think the CPU has an option for reversing the order of the floating point values in the xmm registers.

I checked the version for gdb and found that I was using 6.3 under OS X and 7.5 under Linux, so I checked MacPorts and 7.5 is available, so I installed it.  Then the fun began.  It must be codesigned which requires greating a code signed certificate.  I have done that and signed it still fails to run with this message when I try the run command on a program:


Unable to find Mach task port for process-id 58919: (os/kern) failure (0x5).
 (please check gdb is codesigned - see taskgated(8))

The more I poke at OS X the less I like it.  Installing gdb is trivial under Linux.  Is there some extra permission involved with Linux?  I doubt it, but at least it just works.

Just for grins I did "sudo gdb fpadd" and it now lets the program run, though I get a slew of BFD unknown load command errors.

I will return to this later when I know more.

Well, 1 day later and no closer to understanding this nonsense.  I have managed to run gdb 7.5 which solves the problem with the xmm register values being reversed.  I still have problems running gdb as seyfarth instead of root.  I tried installing a newer version of gcc which does not help.  For my test code I am linking with gcc so it seems possible that some linker options might result in an exe file which gdb recognizes as having a need for avx support.

I suppose I will get this solved, but will I end up with a clearly defined way for people to configure their own Macs to support floating point properly with ebe?  I think I will refrain from entering a complete discussion in the book and refer to a web site so that I can adjust the description as I learn more.

After a couple more days of poking around I have reached a decision.  I have implemented a xmm:reverse ini parameter which will allow someone who is using a "defective" gdb to see the xmm registers properly.  I expect that in a few months OS X will have an easily installable gdb with proper xmm/ymm register support.  I will point out that there is a problem with some versions of gdb in the book and suggest where to look on my web site for further information.

I did find a workable solution using the lldb debugger which comes with xcode.  This would make ebe unnecessarily complex for a few months.  There would be enough changes to make it likely that there would be additional bugs.  I decided to exercise a little patience.  No matter which choice I made I would have to document it in the book.  Using the gdb which came with xcode makes it less trouble to use at the cost of setting a value in .ebe.ini.