dtrace

I had written about Mac dtrace introduced in 10.5. Attended a cool lecture yesterday which covered dtrace introduced in Solaris.

dtrace as the name conveys is dynamic tracing. It allows you to trace the kernel as well as user land processes thus filling in the void by other tools like ktrace and the likes.
dtrace doesn't have any additional overhead. Why? Read on. The dtrace framework is composed of providers - wherein providers are the ones which provide the probes for the various components of the kernel/userland susbsystem. When a dtrace command or a D script (or python/perl/korn script which has dtrace commands) is fired, then the providers which are nothing but loadable kernel modules (LKM) are loaded by the dtrace VM, thus enabling the trace for the particular subsystem. This on demand patching of the kernel ensures that there is no overhead, unless enabled - that too only providers that are demanded for, are loaded.

Now how do you load only the providers that you are interested in?
Well, this does take a little practise/playing with the dtrace command, I guess.. Dtracce has a set of providers, each one tackles a particular subsystem. Predicates allow you to drill down to the level that you want for e.g. to a particular app's particular module's tracing - and that too to the particular module's condition of occurence. (the example that i was introduced to by the speaker was app:firefox, method/module:poll, ts>n seconds and so on.)

All and all, sounds like a cool thing to be armed with. (Been there, done that without dtrace - with ktrace, kdb and weeks of debugging.)

To read more on this:
http://www.sun.com/bigadmin/content/dtrace/dtrace_usenix.pdf
http://developers.sun.com/solaris/articles/power_of_dtrace.pdf