Ctor conflicts

Perhaps the content of this post is trivial and widely known(?), but I just spent some time fixing a bug related to the following C++ behavior.

Let’s take a look at this code snippet:

// main.cpp ------------------------------

#include 

void bar();
void foo();

int main(int argc, const char *argv[])
{
    bar();
    foo();
    return 0;
}

// a.cpp ---------------------------------

#include 

struct A
{
    A() { printf("apple\n"); }
};

void bar()
{
    new A;
}

// b.cpp ---------------------------------

#include 

struct A
{
    A() { printf("orange\n"); }
};

void foo()
{
    new A;
}

The output of the code above is:

apple
apple

Whether we compile it with VC++ or g++, the result is the same.

The problem is that although the struct or class is declared locally the name of the constructor is considered a global symbol. So while the allocation size of the struct or class is correct, the constructor being invoked is always the first one encountered by the compiler, which in this case is the one which prints ‘apple’.

The problem here is that the compiler doesn’t warn the user in any way that the wrong constructor is being called and in a large project with hundreds of files it may very well be that two constructors collide.

Since namespaces are part of the name of the symbol, the code above can be fixed by adding a namespace:

namespace N
{
struct A
{
    A() { printf("orange\n"); }
};
}
using namespace N;

void foo()
{
    new A;
}

Now the correct constructor will be called.

I wrote a small (dumb) Python script to detect possible ctor conflicts. It just looks for struct or class declarations and reports duplicate symbol names. It’s far from perfect.

# ctor_conflicts.py
import os, sys, re

source_extensions = ["h", "hxx", "hpp", "cpp", "cxx"]

symbols = { }
psym = re.compile("(typedef\\s+)?(struct|class)\\s+([a-zA-Z_][a-zA-Z0-9_]*)(\\s+)?([{])?")

def processSourceFile(fname):
    with open(fname) as f:
        content = f.readlines()
    n = len(content)
    i = 0
    while i < n:
        m = psym.search(content[i])
        i += 1
        if m == None:
            continue
        symname = m.group(3)
        # exclude some recurring symbols in different projects
        if symname == "Dialog" or symname == "MainWindow":
            continue
        # make sure a bracket is present
        if m.group(5) == None and (i >= n or content[i].startswith("{") == False):
            continue
        loc = fname + ":" + str(i)
        if symname in symbols:
            # found a possible collision
            print("Possible collision of '" + symname + "' in:")
            print(symbols[symname])
            print(loc)
            print("")
        else:
            symbols[symname] = loc

def walkFiles(path):
    for root, dirs, files in os.walk(path):
        for f in files:
            # skip SWIG wrappers
            if f.find("PyWrapWin") != -1:
                continue
            # skip Qt ui files
            if f.startswith("ui_"):
                continue
            fname = root + os.sep + f
            ext = os.path.splitext(fname)[1]
            if ext != None and len(ext) > 1 and ext[1:] in source_extensions:
                processSourceFile(fname)


if __name__ == '__main__':
    nargs = len(sys.argv)
    if nargs < 2:
        path = os.getcwd()
    else:
        path = sys.argv[1]
    walkFiles(path)

In my opinion this could be handled better on the compiler side, at least by giving a warning.

ADDENDUM: Myria ‏(@Myriachan) explained the compiler internals on this one on twitter:

I'm just surprised that it doesn't cause a "duplicate symbol" linker error. Symbol flagged "weak" from being inline, maybe? [...] Member functions defined inside classes like that are automatically "inline" by C++ standard. [...] The "inline" keyword has two meanings: hint to compiler that inlining machine code may be wise, and making symbol weak. [...] Regardless of whether the compiler chooses to inline machine code within calling functions, the weak symbol part still applies. [...] It is as if all inline functions (including functions defined inside classes) have __declspec(selectany) on them, in MSVC terms. [...] Without this behavior, if you ever had a class in a header with functions defined, the compiler would either have to always inline the machine code, or you'd have to use #ifdef nonsense to avoid more than one .cpp defining the function.

The explanation is the correct one. And yes, if we define the ctor outside of the class the compiler does generate an error.

The logic mismatch here is that local structures in C do exist, local ctors in C++ don't. So, the correct struct is allocated but the wrong ctor is being called. Also, while the symbol is weak for the reasons explained by Myria, the compiler could still give an error if the ctor code doesn't match across files.

So the rule here could be: if you have local classes, avoid defining the ctor inside the class. If you already have a conflict as I did and don't want to change the code, you can fix it with a namespace as shown above.

Qt’s GUI Thread

If you’re a Qt developer, you surely are aware of the fact that you can only display GUI elements and access them from the main thread. This limitation as far as I know is mostly bound to the limitations of X and it isn’t to exclude that multithreading support for GUIs will be added soon.

This limitation never caused me any trouble, since the signal & slots mechanism is thread-safe and communicating between threads and GUI elements can be achieved through it. However, yesterday I needed to show a messagebox in a method and, in case the code is not executing in the main thread, show a native win32 MessageBox instead of a QMessageBox (of course, only on Windows can I do that, on other platforms when I’m not in the main thread, I won’t show anything).

Anyway, here’s a simple method to establish if we’re running the GUI thread:

bool isGuiThread()
{
    if (QCoreApplication::instance()->thread() == QThread::currentThread())
        return true;
    return false;
}

As you can see this is a pointer comparision, but can we rely on the value returned by currentThread? Yes, we can since the pointer is associated with the thread itself as we can see from the code of the method:

QThread *QThread::currentThread()
{
    QThreadData *data = QThreadData::current();
    Q_ASSERT(data != 0);
    return data->thread;
}

// thread_win.cpp

QThreadData *QThreadData::current()
{
    qt_create_tls();
    QThreadData *threadData = reinterpret_cast(TlsGetValue(qt_current_thread_data_tls_index));
    if (!threadData) {
        QThread *adopted = 0;
        if (QInternal::activateCallbacks(QInternal::AdoptCurrentThread, (void **) &adopted)) {
            Q_ASSERT(adopted);
            threadData = QThreadData::get2(adopted);
            TlsSetValue(qt_current_thread_data_tls_index, threadData);
            adopted->d_func()->running = true;
            adopted->d_func()->finished = false;
            static_cast(adopted)->init();
        } else {
            threadData = new QThreadData;
            // This needs to be called prior to new AdoptedThread() to
            // avoid recursion.
            TlsSetValue(qt_current_thread_data_tls_index, threadData);
            threadData->thread = new QAdoptedThread(threadData);
            threadData->deref();
        }

        if (!QCoreApplicationPrivate::theMainThread) {
            QCoreApplicationPrivate::theMainThread = threadData->thread;
        } else {
            HANDLE realHandle = INVALID_HANDLE_VALUE;
#if !defined(Q_OS_WINCE) || (defined(_WIN32_WCE) && (_WIN32_WCE>=0x600))
            DuplicateHandle(GetCurrentProcess(),
                    GetCurrentThread(),
                    GetCurrentProcess(),
                    &realHandle,
                    0,
                    FALSE,
                    DUPLICATE_SAME_ACCESS);
#else
                        realHandle = (HANDLE)GetCurrentThreadId();
#endif
            qt_watch_adopted_thread(realHandle, threadData->thread);
        }
    }
    return threadData;
}

qt_create_tls just calls once for every thread TlsAlloc and if the data for the current thread hasn’t been set yet, it is set with TlsSetValue. So, we can rely on a pointer comparision.

Qt: Now LGPL

Nokia today announced that its Qt cross-platform user interface (UI) and application framework for desktop and embedded platforms will be available under the open source LGPL version 2.1 license from the release of Qt 4.5.

I’ve been waiting for such a decision by Nokia and yet it really came as a surprise. Making Qt free even for closed software will hugely increase their popularity. It will also allow me to develop some crossplatform freeware utilities. I’m still working at the kernel of the CFF Explorer in my free time, but once the kernel is finished it can be used to develop some nice stuff apart from the CFF Explorer itself.

This confirms what I wrote in the article about Qt internals and reversing. This framework will be used more and more in the future.

Kudos to the best framework of all time.

Dynamic C++

As anticipated, I just finished my Dynamic C++ proposal. I’d like to thank my friend Quake2 for his support during the last month.

I wrote this document because I needed to express myself about this subject. Despite the fact that C++ is one of the most used programming languages, especially for serious projects, it gets much criticism for being messy, bloated, complicate etc. I believe these critics miss the point. Yes, C++ is a very powerful programming language and that’s why it is difficult. And this is also why sometimes C++ source codes are poorly written. I don’t believe in improvements of C++ resulting in a new programming language. All attempts in that direction have failed. I think that C++ is here to stay for many reasons. Not only because of the amount of code already available in C++, but also because at the moment there isn’t a better programming language for large projects. The only thing I want is for C++ to evolve, but not by losing compatibility with older code or by removing some features. No, I’d like C++ to evolve in a healthy and compatible way. This paper contains the suggestions to achieve this and I will demonstrate technically how it can be implemented at low level.

Everybody should be warned that the material contained in this paper is purely theoretical. The first idea behind this paper came out while working on a particular project. At the time I discovered myself in need of particular dynamic features. So, for many months I had some ideas in the background of my mind and decided eventually to write them down. So, in this paper I’m going to talk about the current status of dynamism for C++, why dynamism is important and what could be done. At the time of writing (November 2008) the new C++0x (or C++09) standard has not yet been introduced. However, I will talk about it throughout this paper when the related topic is affected by it.

Qt Internals & Reversing

Today I took a break from the larger article I’m currently writing. To relax, I wrote a smaller article about the Qt framework. I hope you enjoy.

Qt Internals & Reversing

Half of the text of this article comes from my larger paper “Dynamic C++ Proposal”. I decided that it was useful to take the part about Qt internals, put it into another article and extend it by adding a reversing part. Because of its nature, this is not the usual kind of article I write. In fact, I wrote the reversing part in less than a day. So, this is a very easy one. However, I think it is useful for people who need to reverse a Qt application and certainly wouldn’t consider reading my other paper about Dynamic C++, which doesn’t sound like a paper about Qt and, in fact, isn’t a paper about Qt: the paragraph about Qt is only one among many others. Moreover, I haven’t seen serious articles about this subject.

The first thing which needs to be considered when reversing Qt applications is what Qt brought to the C++ language. Events (inside the Qt framework) are just virtual functions, so nothing new there. This is not a C++ reversing guide. What is new in Qt are signals and slots, which rely on the dynamism of the Qt framework.

So, first thing I’m going to show how this dynamism works. The second part focus on reversing and, at that point, I will show how to obtain all the metadata one needs when disassembling a “Q_OBJECT” class.