Displaying stack frames in ebe

It is now possible to display the current function's relevant stack data in a stack frame window in ebe.  To do this use the new frame macro in the start of a function.  For example let's suppose you want to use 3 local variables in your main function and call some function with 4 parameters.  Then you should start main like this

main:
      push   rbp
      mov    rbp, rsp
      frame  2, 3, 4   ; main has 2 parameters, will use 3 locals, and
                       ; will call functions with a maximum of 4 parameters
      sub    rsp, frame_size
 
The frame macro does not generate any instructions.  However it does determine frame_size, which could be 0 in functions with no local variables.  Ebe keeps track of the lines after the frame macro and builds a table in the stack frame window from that point forward in the code.  It will be accurate only after the subtraction of frame_size.

With Linux and OS X the frames are generally pretty simple.  Also the use of 6 register parameters means that a lot of functions will require no space on the stack for parameters to the current function or to call functions.

Here is the stack from using 2, 3, and 4 for Linux:

You can see that local1 is on the third line of the table and it addressed using [rbp-8].  Alternatively you can use simple [rbp+local1].  Let's assume you want to save the 2 parameters to main in locals 1 and 2:

      mov    [rbp+local1], rdi
      mov    [rbp+local2], rsi
 
You may have noticed the last row in the table which could have been label4.  This space is there so that the functions which you call can keep the stack on 16 byte boundaries.  The value of frame_size will always be a multiple of 16.

Now suppose you have a function which receives 9 parameters, has 2 locals and the maximum number of parameters for called functions is 5.  Then the current function will have 3 parameters on the stack.  The frame call would be

      frame  9, 2, 3

The stack frame for this function would look like this under Linux or OS X.

 

Here we see an empty row at the top of the table which the calling function had to prepare for.  Before making the call it placed parameters 7, 8 and 9 on the stack as shown as currPar7-9.  You can also see the addressing for parameter 7 is [rbp+16] or more simply stated as [rbp+currPar7].

Similarly you can prepare to call a function with 9 parameters from a function with 2 locals and 2 parameters using

      frame  2, 2, 9

This will generate a stack frame like this one for Linux and OS X

Again you see an empty row in the table above newPar9 so that rsp will remain a multiple of 16.  You can place values into these stack-based parameters using [rsp+newPar7].  Note that the recommendation is to use rsp for the new parameters since they are basically at positive offsets from rsp.

The main goal for ebe is to make assembly programming easier without sacrificing the main benefit from learning assembly which is to learn how the computer works.  This explains why I chose to not implement a function macro which could have generated the standard stack frame code.  If you really want it to be easier, maybe C is the answer.

I have also been working on some register re-naming tricks which can make it easier to write more complex functions, but that discussion will be for another day.

Linux Binary Compatibility

I have had some issues with lack of binary compatibility of Linux.  I can build ebe using a lot of shared libraries and if people have those libraries installed it is highly likely that the binary will work properly.  Their computer would need matching versions of a moderately large collection of .so files.

To avoid requiring that ebe users install a bunch of libraries I have developed a script named "build_package" which uses ldd to determine the shared objects used by ebe and it builds a directory named "ebe.d" with ebe copied in as "ebe.exe" and all the shared objects.  The idea is that on the target system ebe would be started using a shell script named "ebe" which sets LD_LIBRARY_PATH to ebe.d allowing all the shared libraries from my system to be used.

This works with the same version of Ubuntu (13.04) for me, but fails on 12.04.  Interestingly on my virtualbox 12.04 system ebe.exe works if I allow all the libraries to be loaded normally.  So I tried to match the strategy of firefox which is a fairly successful program.  It is loaded by a script though it does not use LD_LIBRARY_PATH.  Firefox does use some special shared libraries but these apparently are loaded using dlopen rather than automatically.  They don't appear when you use ldd.

So I started removing shared libraries.  After removing a few of them I started getting incompatible messages about Qt versions.  Apparently Qt knows that the Qt shared libraries I copied to 12.04 are a different version  from the libraries I installed previously.  I did not find an environment variable which pointed to the older libraries, so I assume that Qt does some form of internal consistency checking.  This may be something I can get around, but I don't have a clue about how to dodge this test now.

I considered briefly forcing ebe users to install from source, but the requirements are fairly large.  I was trying to make ebe accessible to novices.  Novices shouldn't have to face a long list of instructions just to get their IDE installed.  I had the experience of installing a copy of ebe modified to use Qt5 recently.  It took probably an hour of piddling around to figure out the packages required to get ebe to build.  Maybe I missed the obvious qt5-dev package, but I tried to find the obvious.  I wouldn't call myself a novice though I know I can make some really stupid mistakes.

So I am left with providing a separate install package for each of several versions of Linux.  What a waste of effort!  I am sure there is a better way, but I have spent more time looking for a better way than  it would take me to provide 3 versions of my install package for a year.

To reduce the effort I will start using rsync to update my downloadable files on sourceforge.  I had been using their web interface on each computer.  I will still have to do a fair amount of work on each revision, so I need to make less frequent updates to the binary installers.  I can still push source code changes regularly since that is so easy, but the installers I will try to update perhaps once a month.

I also tried to use an authorized_keys file on sourceforge to make it quicker to do the rsync, but that didn't work.  It didn't really matter since it takes a few seconds to type in my password and an hour or two to prepare the binary packages on about 5 computers.

Using Qt Linguist to Translate EBE's Messages to Multiple Languages

Over the past week I have been using Qt's language translation facility to translate all the words and phrases used in ebe into multiple languages.  Qt is designed for easy use of multiple languages.  If you make a good habit of using their tr function with every string, then retrofitting languages is fairly easy.

Consider the use of QMessageBox::warning to present warning messages.  Here is one of my calls:

    QMessageBox::warning(this,tr("Error"),
       tr("The first index can't be\n greater than the last."),
       QMessageBox::Ok, QMessageBox::Ok);

You see that there are 2 calls to tr in the call to the warning function.  If you have not implemented any other languages, tr("Error") will return "Error".  If you have set up your program with other languages, then tr will use the active language to rapidly find a translation for "Error".  In the case of French, it might return "Erreur".  So for each language you must prepare a translation for each string embedded in a call to tr.

The file which contains translations is an XML with the extension "ts".  You need to a TRANSLATIONS variable to your Qt project file (ebe.pro for ebe).  Here is my current value

    TRANSLATIONS = ebe_fr.ts ebe_sp.ts ebe_sv.ts \

                   ebe_de.ts ebe_pt.ts ebe_hi.ts \

                   ebe_zh.ts ebe_ru.ts ebe_ar.ts \

                   ebe_bn.ts ebe_in.ts ebe_ja.ts

The project file is processed by the lupdate program to determine all the translation files and also the location an content of each tr call.  Each of the files named has a 2 character language code like "fr" for "French" which is used by the Qt lupdate program to determine the language.  If a particular .ts file does not exist it will be created with an empty translation for each tr string.  If a string is repeated within a source file, it exists as one entry in the .ts file with the line numbers for each occurrence listed with the string.  If a file already exists it is updated with changed information while retaining existing translations.

The translation process consists of replacing the empty translations with reasonable strings from the appropriate language.  This can be done with an editor (provided the editor allows entering the proper characters and accents for the language).  A better choice is to use the Qt Linguist program.  This allows you to move through the .ts file without the bother of keeping the XML format straight.  With Linguist you can type in translations or copy them from other sources.  In my case I have been using Google Translate, though many editors will allow editing in multiple languages.

So far I have translations for French, German, Hindi, Arabic, Swedish, Russian, Chinese, Spanish and Portuguese.  You can see from the TRANSLATIONS variable that I anticipate translations for Bengali, Indonesian and Japanese.  At that point I will have covered the top 10 most used languages of the world with a start toward covering more of Europe.

My main problem is the inconsistent quality of Google Translate's translations.  I really need knowledgeable people to repair my broken translations.  I have had a volunteer from Canada repair my French file and I have volunteers working on Spanish, Portuguese, Hindi, Arabic and Chinese.  I need help with Russian, Indonesian, Bengali, Japanese, German and Swedish.

In general this points out to me the need to recruit volunteers to the ebe project.  I desperately need help with documentation.  I need to complete HTML files detailing how to use ebe.  Ultimately it would be wonderful to have all the HTML files translated into the dozen or so languages selected for ebe.  There are also needs to write sample code and HTML files in the ebe library.  Finally ebe is not nearly perfect and I need a few good programmers to add features and work on the ebe C++ code.  This is an open source project so it needs volunteers.

Long term changes include adding a function/class database to provide call information within the editor and adding lessons to the system.  The goal is to make ebe into a great tool for teaching and lessons seem important.  I would also welcome any bright ideas which could further the cause.

Syntax highlighting for Fortran and Assembly

I finally got around to writing classes for Fortran and Assembly source code highlighting.  It was fairly easy to convert the existing Highlighter class into a base Highlighter class and a derived CppHighlighter class.  Then I copied the code to produce FortranHighlighter and AsmHighlighter.

I found a collection of Fortran 20xx keywords online which I used to replace the C++ keywords in the constructor for FortranHighlighter.  Fortran essentially throws away white space so keywords like "end do" become "enddo".  For my convenience I generated the keywords "end", "do" and "enddo" to make it work properly.  Then there was a minor bit of coding to handle Fortran comments and strings properly.

I had previously stored all the x86-64 instructions in src/assembly/instructions and this file is read into a set of strings named instructions.  So I only had to change the test for keywords to be testing for a string in the set to manage keywords properly for AsmHighlighter.  I had to simplify the comment code a little since Assembly comments all begin with ';' and go to the end of the line.  I left the string handling as it was.

As I did previously I implemented state machines for lexical analysis using gotos.  It is so easy and pretty clear when done nicely.   Now I have 70 gotos in highlighter.cpp.  It's a cheap thrill.

Coping well with the lack of line numbers

One of my goals in the Qt version of ebe is to transparently support OS X assembly language as well as Linux assembly language.  There are several basic problems with using yasm under OS X.

• OS X uses rip-relative addressing.
• Global functions use an underscore prefix.
• There is no debug information provided by yasm for use in gdb.

The first two problems are fairly easy to cope with.  First the rip-relative addressing only matters when you attempt to using indexing of an array or accessing a structure component in the data segment.  For those cases if you use load-effective-address to get the address of the array or struct into a register, this works on both Linux and OS X.

The second problem can be solved using macros.  I have prepared a set of macros which I automatically prefix each assembly file (using yasm's -P option).  The macros add "default rel" under OS X to establish rip-relative addressing and translate each of about 350 function names to have prefixes including main, scanf, printf, ...   So the source code can use main without worrying about the need for an underscore.  There is also a cname macro which can turn any name into a macro which will have an underscore prefix under OS X and not under Linux.  So the set of macros takes care of the first 2 issues.

The lack of debugging support is not as total as it could be.  You can still find globals and addresses using nm and within gdb, but there is no way to set a breakpoint by using a line number and when gdb stops after a next instruction command it won't tell you the next line of the function to execute.

My original solution to this was to inspect the listing file and determine relative addresses for each line and then query by either gdb or nm to determine an actual address such as &main.  This is a fair amount of code and more code means a higher probability of error.

My next solution to the lack of line numbers is to make my own.  I am now generating a debug asm file from the original with each original line preceded by a generated label with a line number in the label.  To properly handle local labels, my generated labels need to be local labels, so I need to create a global label to stick at the start of the file.

So a file would start with

ebe_debug;           ; generated global label
.filename_line_1:    ; generated local label with a line number
...                  ; whatever was originally on line 1
.filename_line_2:
...                  ; whatever was originally on line 2

So with a file with 100 lines there would be 201 lines in the debug asm file and the generated instructions are the same.  Now it is possible to extract all the address information using nm for the executable and issue breakpoint commands like:

break main.filename_line_8

By choosing a nice solution I managed to get a bonus: gdb reports the location when it finishes a command with something ending in "in main.filename_9", so I can readily determine the filename and the line number for the next line to execute and highlight it in ebe.

This solution was fairly good.  It was simple, but it interfered with using macros among other issues.  For repeat macros there would be multiple occurrences of the same label.  After using this solution for a while, I reverted back to the solution requiring analysis of the listing file to determine relative addresses which are later translated to program addresses using addresses of globals in the program.  This worked smoothly.

I think the total solution might require identifying labels well.  Unfortunately yasm allows labels with or without colons.  With a colon a label is obviously a label.  Without a colon an instuction looks about the same.  I rounded up a relatively complete collection of x86_64 instructions and yasm pseudo-ops to store in a QSet<QString> to identify when the first word on a command is an instruction or a label.  There are roughly 1600 names in the set.  It is a truly arcane instruction set, but fortunately you can learn a fairly small subset and do a fairly good job.

Left to deal with are handling data items with global and local labels.  I need to review this again to see if I really must identify the labels in the source.  So far I have identified the range of each global label which would determine which global label is appropriate for a variable identified by a local label.  I think this is necessary, but I hope to simplify this too.