User-space C++ application tracing on Fedora using Systemtap

Ever since DTrace came out I realized how much that was what I really wanted to do whenever my program was doing something other than what I assumed it was doing. "Tracing" -- I want that.

Debugging is fine; but you need to have debug symbols, you need to pick the right spot in the code to break at, you have to keep manual track if you're tracing a long-running execution, and real-time applications may not play nice when paused for arbitrary periods. Often I would want to not follow a thread of execution up and down its call stack -- rather I'd be interested in the "cross-cutting concerns" -- how a certain variable changes over time.

DTrace will never be coming to my favorite platform (GNU/Linux), but there is a similar library called SystemTap for Linux, which has had the Linux kernel instrumented for a while, but just recent has allowed tracing in user-space programs. I think you will need Fedora 14 to run user-space traces; at least that is what I am using. With the appropriate setup you can start tracing your own C++ applications without modifying the source.

SystemTap uses "static" (preprocessor macro-based source code) probes that are compatible with DTrace; so if you decide to mark-up your source with these macros (as Java, Python, MySQL, et al. are) you can use DTrace or SystemTap depending on your platform. These probes are turned into no-ops when not in use, so they are very fast. However the usage that I am interested in is ad-hoc tracing for the purpose of debugging or simple profiling. Thankfully for this case all you need is debug symbols.

Once you have SystemTap installed, and your program compiled with symbols, you can write your custom probes. The syntax of the probes is beyond our scope, but you can read about it here. Instead I will post a simple "hello world" demo as follows:

#include < iostream >
#include < vector >

using namespace std;

namespace Baz
{
    struct Foo
    {
        int i; int n; float f;

        Foo() : i (0), n (0), f (3.14) {}
        Foo(int a, float b) : i (a), n (0), f (b) {}
        Foo(Foo const &c) : i (c.i), n (c.n), f (c.f) {}
        ~Foo() {}

        int dolart () { f += i; f /= 3.14; return 0; }
        int dofizz (int a) { n = a; i += n; return n; }
    };
}

int main()
{
    vector < int > test;
    vector < Baz::Foo > list;

    for (int i=0; i<5; ++i)
    {
        test.push_back (i);
        list.push_back (Baz::Foo (i, 1.1));
    }

    vector < Baz::Foo >::iterator i = list.begin();
    vector < Baz::Foo >::iterator e = list.end();

    for (int n=0; i != e; ++i)
        i->dolart (), i->dofizz (++n); 

    return 0;
}

And if we use the following "tapset" on the code

global lart_count
global fizz_count
global push_back_count

probe process("a.out").function("Baz::Foo::do*").return 
{ if ($return == 2) printf("Foo returned 2!\n"); }

probe process("a.out").function("Baz::Foo::dolart") 
{ ++lart_count; }

probe process("a.out").function("Baz::Foo::dofizz") 
{ ++fizz_count; }

probe process("a.out").function("vector < * >::push_back")
{ ++push_back_count; }

probe process("a.out").function("main").return
{
  printf("dolart was called %d times.\n", lart_count);
  printf("dofizz was called %d times.\n", fizz_count);
  printf("vector::push_back was called %d times.\n", push_back_count);
}

And run it as follows:

$ g++ -g test.cpp
$ sudo stap probe.stp -c ./a.out

Then we should get the following result:

Foo returned a 2!
dolart was called 5 times.
dofizz was called 5 times.
vector::push_back was called 10 times.

A couple issues I noticed

• You must run this as root. This is because stap has to compile and load a kernel module that has enormous power to probe your system.
  
• Seems to run a bit slow on the simple example. This could be one-time start-up due to compiling a kernel module, or it could be a real-time overhead of not using the cheaper static probes.
• It seems to spew a lot of strange mangled C++ names for an unknown reason.
• Constantly respecifying process("a.out") seems redundant. There's probably a way to get around this overhead.
  

State of the Driver Stack in Linux

Excellent article! Surprising find.

How to get GStreamer codecs working on Moblin 2.0 beta

Since the latest Moblin is based on Fedora 10, we can install gstreamer-plugins-bad over yum to get most codecs working correctly. However, we'll have to hack things somewhat.

Moblin uses the OpenSSL from F11, but libstdc++ 4.3 from F10. Since some media libraries seem to rely on both we will have to try to mix repositories carefully.

1. download the latest rpmfusion and fedora release rpms, as they contain all the yum repository configurations
2. force install rpmfusion release:
   • $ rpm -Uvh --nodeps rpmfusion*release*.rpm
     
3. unpack fedora release:
   • $ rpm2cpio fedora-release.rpm | cpio -idv
   • $ cp ./etc/yum.repos.d/* /etc/yum.repos.d
4. manually set configurations
   
• $ sed -i 's/$releasever/11/' /etc/yum.repo.d/rpmfusion*
  
• $ sed -i 's/$releasever/10/' /etc/yum.repo.d/fedora*
• $ sed -i 's/enabled=1/enabled=0/' /etc/yum.repo.d/fedora*
• $ sed -i 's/gpgcheck=1/gpgcheck=0/' /etc/yum.repo.d/fedora*
5. install dirac-libs from F10
   • $ yum install --enablerepo=fedora,updates dirac-libs
6. change to F11
   
• $ sed -i 's/10/11/' /etc/yum.repo.d/fedora*
• $ rm -rf /var/cache/yum
7. install the rest
   • $ yum install --enablerepo=fedora,updates --exclude=dirac-libs gstreamer-plugins-bad gstreamer-plugins-flumpegdemux gstreamer-ffmpeg
     
   
   

Why Windows is not yet ready for the Desktop

I don't spend my time telling other people which OS should or shouldn't suit their way of working. But it seems there are people who do, and like to get blog hits for it.

The problem with these "critiques" is always that the author is carrying around the self-serving assumption that their preferred OS embodies the only real way to organize a software ecosystem, and all others have inferior value. Moreover, since they are naturally only looking for a way to justify their existing pre-conclusion, they are often sadly misinformed about most of their "complaints", half of which are either entirely subjective, or just flat-out wrong.

And it is thus that I find myself moved enough to mock their contribution to the state of public discourse as follows (public service announcement: this is tongue-in-cheek parody):

Preface:

In this document we only discuss Windows deficiencies while everyone should keep in mind that there are areas where Windows has excelled other OSes.

A primary target of this comparison is Linux OS.

Windows major shortcomings and problems:

0. Premise: free and open software will stay indefinitely. Full stop. You may argue eternally, but free software is the ultimate disruptive technology, moving up from the low ground, replacing complicated and ill-fitting proprietary alternatives at every turn, such as web-browsers, e-mail clients, video players, office software, etc., which at one point cost money, but now most people find that they can no longer justify spending money to buy an upgrade for more "Clippy the Happy Assistant". Proprietary software will only be able to stay relevant by searching out ever more niche applications, or by massive expenditure on research in high-end applications for which it will take time for the ideas and algorithms to filter down to the greater community, and thus a brief window of profitability will remain. Software patents are nothing but a destructive force to retard innovation, and with more and more of the technology and legal communities realizing this basic fact, software patents are about to go away forever.

1. Security

1.1 History's greatest playground for malicious software. With unpatched machines on the internet taking only minutes to become infested with viruses, or become a slave bot for massive illegal spamming operations, Windows is a blight on the Internet's infrastructure.

1.2 Countless applications are released every year with obvious security holes. The programmers that make Windows applications are clearly some of the worst.

1.3 Microsoft has countless times avoided appropriate steps to secure the OS and limit the potential damage a compromised binary could cause. It has consistently either or added half-measures, out right refused to take necessary steps to ensure a safer computing environment for all users, for fear of making "Auntie Jo'" 10% more confused about the "1.3GHz hard drive" on her desk.

1.4 Every windows application I've ever installed messes with the Registry, places files about my hard drive which it never cleans up, installs icons, or worse surreptitiously installs spy- or ad-ware.

1.5 Any OS that regularly requires a wipe and reinstall to fix is beyond tolerance by any sane person.

1.6 A galore of software bugs across all applications. Just look into Vista, or call Microsoft tech support, pay exorbitant support fees, then wonder why some bugs are now ten years old with over several dozens of duplicates and no one is working on them.

2. User Interface

2.1 No consistent API. Win32? MFC? WinForms? WPF?

2.2 No scripting bindings for UI programming. No Python, Perl, Ruby, Java, etc.

2.3 Themeing and skinning support is laughable. Widget toolkit, display, rendering, input, and window managers, are all joined in a ridgid, monolithic blob, opaque to outside developers. Non-trivial changes to look and behaviour of the UI require either proprietary add-ons or third-party hacks; and even then most of your choices are hard-coded by Microsoft designers.

2.4 Lack of CLI (command line interface) errors for user applications (see clause 4.). All GUI applications should have a CLI errors presentation. Why on earth would you flash some crazy warning message to the user when you should be logging it to a file for a skilled technician to view instead of the poor unsuspecting end-user.

3. Interoperability

3.1 Windows has NO interoperability with non-Windows OSs. Installing Windows arrogantly destroys any previous OS boot-loader you may have had. Totally unable to read non-FAT or NTFS partitions.

3.2 Windows ships no other runtime environments except .NET. Has actively tried to disable or cripple competing platforms such as Netscape or Java.

3.3 Microsoft is in regular legal trouble for monopolistic and anti-competitive practices, which as a consumer of non-Microsoft products, means Microsoft considers me an enemy. Why own an OS that is constantly out to defeat you, from a vendor that requires massive anti-trust lawsuits to force it to simply not behave in an under-handed manner?

3.4 It should be possible to configure everything from the command line. Why should I give myself a work-place injury clicking everywhere with the mouse like a tweaking junkie in order to make a change that could be described succinctly in a line or two of text?

4. Drivers

4.1 Windows driver support is so abysmal, each individual device manufacturer must ship drivers with the device itself. If you have to reinstall windows, none of your devices will work until you individually download and install the latest versions from each vendor's website, potentially consuming many long frustrating hours.

4.2 Drivers often need to be installed, tweaked, or configured before they can even be used as intended. They often don't work "out of the box". Moreover, they never seem to be *just* drivers, there is always some application that gets installed without your consent which provides questionable value yet consumes resources and slows your computer down.

4.2 Drivers are one of the main sources of system instability (likely just behind viruses/malware). Poor quality drivers make Windows experience painful.

4.3 Windows has no means to reliably update drivers when critical updates have been made available for them.

4.4 A lot of Linux specific embedded devices do not have any Windows support. An argument that embedded device developers should make their device Windows compatible is silly since that way Windows won't ever gain even a traction of popularity among people who need source-level access to the OS. Why should I install an OS where my own hardware doesn't work?

5. Installing Applications

5.1 Very few Windows applications, by volume, are free or open source; which means you are totally beholden to application developer in ways that would never be allowed by law for makers of physical products. Happen to have your business critical data in a proprietary format when your license runs out? Lost your dongle just before the big presentation? Had to transfer your application to another computer because your laptop was stolen? Sorry to hear you just went out of business.

5.2 Windows has no regular time-based release cycle. You paid good money for a few features and a lot of bugs. It may be a few years from now when you can expect them truly fixed, but you can't count on it. And you'll have to pay again.

5.3 Windows has no central means of downloading new software, their dependencies, or upgrades. Each new application must be purchased from a physical store, or from each individual vendor's website. There is no dependency tracking (or worse no library sharing!), and updating for security, bug-fixes, or features is ad-hoc and entirely dependent on the whim of the vendor. Likely the vendor will use remote updating features to unethically sneak updates to your computer without your knowledge.

5.4 Windows comes almost barren on a fresh install. To get your machine back to a usable state, you must spend hours remembering what applications you had installed, and manually downloading and installing each one individually. With a reboot in between each install.

5.5 Windows applications need to reboot any time a new application or library is installed. 1991 called. They want their loading technology back. I hear DLL-hell isn't a problem any more though.

5.6 Microsoft enforces a great many intra-windows compatibility constraints to minimize the ever-present costs of portability, but it comes at the cost of inconsistent behaviour, buggy programs, and internal complexity that is slowing rotting Windows itself from the inside out.

5.7 Lack of hard-core Linux programs like grep/awk/GDB/valgrind/SystemTap/SELinux. Programmers just won't bother installing Windows until they can work for real.

6. Problems stemming from the fact that Windows isn't Linux

6.1 Ok I am officially tired of this game.

To be clear, I don't necessary truly believe all of the above, as unlike most people, I realize the world is full of complications and subtlety -- I'm just tired of hearing coming in the opposite direction, and had to vent lest my head explode from idiocy-overload.

I just got the new HP C5180 all-in-one printer I needed to apply for my passport. It works perfectly in Linux -- I just plugged it into the network via ethernet, set up CUPS, and got a perfect result.
very recommended.