Using Qt Linguist to Translate EBE's Messages to Multiple Languages

Over the past week I have been using Qt's language translation facility to translate all the words and phrases used in ebe into multiple languages.  Qt is designed for easy use of multiple languages.  If you make a good habit of using their tr function with every string, then retrofitting languages is fairly easy.

Consider the use of QMessageBox::warning to present warning messages.  Here is one of my calls:

    QMessageBox::warning(this,tr("Error"),
       tr("The first index can't be\n greater than the last."),
       QMessageBox::Ok, QMessageBox::Ok);

You see that there are 2 calls to tr in the call to the warning function.  If you have not implemented any other languages, tr("Error") will return "Error".  If you have set up your program with other languages, then tr will use the active language to rapidly find a translation for "Error".  In the case of French, it might return "Erreur".  So for each language you must prepare a translation for each string embedded in a call to tr.

The file which contains translations is an XML with the extension "ts".  You need to a TRANSLATIONS variable to your Qt project file (ebe.pro for ebe).  Here is my current value

    TRANSLATIONS = ebe_fr.ts ebe_sp.ts ebe_sv.ts \

                   ebe_de.ts ebe_pt.ts ebe_hi.ts \

                   ebe_zh.ts ebe_ru.ts ebe_ar.ts \

                   ebe_bn.ts ebe_in.ts ebe_ja.ts

The project file is processed by the lupdate program to determine all the translation files and also the location an content of each tr call.  Each of the files named has a 2 character language code like "fr" for "French" which is used by the Qt lupdate program to determine the language.  If a particular .ts file does not exist it will be created with an empty translation for each tr string.  If a string is repeated within a source file, it exists as one entry in the .ts file with the line numbers for each occurrence listed with the string.  If a file already exists it is updated with changed information while retaining existing translations.

The translation process consists of replacing the empty translations with reasonable strings from the appropriate language.  This can be done with an editor (provided the editor allows entering the proper characters and accents for the language).  A better choice is to use the Qt Linguist program.  This allows you to move through the .ts file without the bother of keeping the XML format straight.  With Linguist you can type in translations or copy them from other sources.  In my case I have been using Google Translate, though many editors will allow editing in multiple languages.

So far I have translations for French, German, Hindi, Arabic, Swedish, Russian, Chinese, Spanish and Portuguese.  You can see from the TRANSLATIONS variable that I anticipate translations for Bengali, Indonesian and Japanese.  At that point I will have covered the top 10 most used languages of the world with a start toward covering more of Europe.

My main problem is the inconsistent quality of Google Translate's translations.  I really need knowledgeable people to repair my broken translations.  I have had a volunteer from Canada repair my French file and I have volunteers working on Spanish, Portuguese, Hindi, Arabic and Chinese.  I need help with Russian, Indonesian, Bengali, Japanese, German and Swedish.

In general this points out to me the need to recruit volunteers to the ebe project.  I desperately need help with documentation.  I need to complete HTML files detailing how to use ebe.  Ultimately it would be wonderful to have all the HTML files translated into the dozen or so languages selected for ebe.  There are also needs to write sample code and HTML files in the ebe library.  Finally ebe is not nearly perfect and I need a few good programmers to add features and work on the ebe C++ code.  This is an open source project so it needs volunteers.

Long term changes include adding a function/class database to provide call information within the editor and adding lessons to the system.  The goal is to make ebe into a great tool for teaching and lessons seem important.  I would also welcome any bright ideas which could further the cause.

Syntax highlighting for Fortran and Assembly

I finally got around to writing classes for Fortran and Assembly source code highlighting.  It was fairly easy to convert the existing Highlighter class into a base Highlighter class and a derived CppHighlighter class.  Then I copied the code to produce FortranHighlighter and AsmHighlighter.

I found a collection of Fortran 20xx keywords online which I used to replace the C++ keywords in the constructor for FortranHighlighter.  Fortran essentially throws away white space so keywords like "end do" become "enddo".  For my convenience I generated the keywords "end", "do" and "enddo" to make it work properly.  Then there was a minor bit of coding to handle Fortran comments and strings properly.

I had previously stored all the x86-64 instructions in src/assembly/instructions and this file is read into a set of strings named instructions.  So I only had to change the test for keywords to be testing for a string in the set to manage keywords properly for AsmHighlighter.  I had to simplify the comment code a little since Assembly comments all begin with ';' and go to the end of the line.  I left the string handling as it was.

As I did previously I implemented state machines for lexical analysis using gotos.  It is so easy and pretty clear when done nicely.   Now I have 70 gotos in highlighter.cpp.  It's a cheap thrill.

Coping well with the lack of line numbers

One of my goals in the Qt version of ebe is to transparently support OS X assembly language as well as Linux assembly language.  There are several basic problems with using yasm under OS X.

• OS X uses rip-relative addressing.
• Global functions use an underscore prefix.
• There is no debug information provided by yasm for use in gdb.

The first two problems are fairly easy to cope with.  First the rip-relative addressing only matters when you attempt to using indexing of an array or accessing a structure component in the data segment.  For those cases if you use load-effective-address to get the address of the array or struct into a register, this works on both Linux and OS X.

The second problem can be solved using macros.  I have prepared a set of macros which I automatically prefix each assembly file (using yasm's -P option).  The macros add "default rel" under OS X to establish rip-relative addressing and translate each of about 350 function names to have prefixes including main, scanf, printf, ...   So the source code can use main without worrying about the need for an underscore.  There is also a cname macro which can turn any name into a macro which will have an underscore prefix under OS X and not under Linux.  So the set of macros takes care of the first 2 issues.

The lack of debugging support is not as total as it could be.  You can still find globals and addresses using nm and within gdb, but there is no way to set a breakpoint by using a line number and when gdb stops after a next instruction command it won't tell you the next line of the function to execute.

My original solution to this was to inspect the listing file and determine relative addresses for each line and then query by either gdb or nm to determine an actual address such as &main.  This is a fair amount of code and more code means a higher probability of error.

My next solution to the lack of line numbers is to make my own.  I am now generating a debug asm file from the original with each original line preceded by a generated label with a line number in the label.  To properly handle local labels, my generated labels need to be local labels, so I need to create a global label to stick at the start of the file.

So a file would start with

ebe_debug;           ; generated global label
.filename_line_1:    ; generated local label with a line number
...                  ; whatever was originally on line 1
.filename_line_2:
...                  ; whatever was originally on line 2

So with a file with 100 lines there would be 201 lines in the debug asm file and the generated instructions are the same.  Now it is possible to extract all the address information using nm for the executable and issue breakpoint commands like:

break main.filename_line_8

By choosing a nice solution I managed to get a bonus: gdb reports the location when it finishes a command with something ending in "in main.filename_9", so I can readily determine the filename and the line number for the next line to execute and highlight it in ebe.

This solution was fairly good.  It was simple, but it interfered with using macros among other issues.  For repeat macros there would be multiple occurrences of the same label.  After using this solution for a while, I reverted back to the solution requiring analysis of the listing file to determine relative addresses which are later translated to program addresses using addresses of globals in the program.  This worked smoothly.

I think the total solution might require identifying labels well.  Unfortunately yasm allows labels with or without colons.  With a colon a label is obviously a label.  Without a colon an instuction looks about the same.  I rounded up a relatively complete collection of x86_64 instructions and yasm pseudo-ops to store in a QSet<QString> to identify when the first word on a command is an instruction or a label.  There are roughly 1600 names in the set.  It is a truly arcane instruction set, but fortunately you can learn a fairly small subset and do a fairly good job.

Left to deal with are handling data items with global and local labels.  I need to review this again to see if I really must identify the labels in the source.  So far I have identified the range of each global label which would determine which global label is appropriate for a variable identified by a local label.  I think this is necessary, but I hope to simplify this too.  

Retirement == Freedom

I have been retired for 7 or 8 months now and I think I am starting to get used to the new life.  Recently I have started feeling more energetic about programming.  It is different now.  In the past I was either working on a project with a specific goal in mind or generally too busy to focus on new programming projects.

Now don't get me wrong I have done some interesting, fun things as parts of several projects during the last 5 years at USM.  The difference is one of degree.  Over the years I would explore interesting things for fun, but the unpaid projects were usually somewhat small.  Now whatever I decide to do is my choice and I can generally decide when to work on it.

"Toy Box" for ebe

Since about November 19, 2012 I have been working steadily rewriting the ebe integrated development environment in C++ using the Qt toolkit for the GUI controls.  The new ebe is better in many ways than the older one which was written in Python.

One feature I have been considering was to allow beginners to enter simple expressions and evaluate them the same way that the compiler would do.  I searched for C and C++ interpreters thinking that an interpreter might make a good choice for speedy computations.  During my search I came across a Python program which does live calls to g++ to "interpret" the C++ as it is entered.  I had to wonder about the speed of the compilations.

I started by compiling hello.c and hello.cpp to see how long it took.  On my Core i7 desktop it took about 0.1 seconds for compiling hello.cpp which hello.c took about 0.06 seconds.   I copied hello.c to hello.cpp and it still required 0.06 seconds compiled with g++.  That meant that using printf would be superior for my needs over cout.  Compilations could be done in 0.6 seconds and executing the small program took about 0.002 seconds.  So perhaps I could get 16 compilations per second.

I knew enough about the C++ typeid to figure out the type of an expression, but in order to capture the data precisely I wanted to dump the data for the expression as hexadecimal.  That would allow me to capture floats and doubles exactly as they were computed in the expression.  It seemed a shame to write one program to determine the type of an expression and a second program to dump the value of the expression in hex.  So I kept searching.

Eventually I ran across the g++ typeof operator.  This would allow me to declare a variable of the exact type as an expression.   Consider the declaration below
     typeof((a+b+c)/3.0) x;
This declaration determines that the type of the expression is double and declares x as a double.  With this interesting feature it was easy to write code to declare a variable of the right type to assign the value of an expression and then, using sizeof, I could dump each byte of the variable.

Here you can see the "Toy Box" after defining 3 int variablees and evaluating some expressions.  Line 1 of the lower table shows the original type, format and result for the expression in column 1.  In the others I have used the Format combobox to select an alternative format.  This seems like a useful tool to use when learning a language.  I feel sure that a lot of teachers would enjoy illustrating how C/C++ evaluates expressions using the toybox.  This is far superior to using an interpreter which would probably approximate the syntax and the behavior,  This tool uses the compiler so what you see is what you get.