Recently, I had to find bottlenecks in one of our applications that does all sorts of things related to DVB and is handling real-time reading and sending of data streams over IP with bitrates up to 80 Mb/s (in our lab; in real life maybe even more). Our customer created a configuration which essentially brought the app to the crawl and I had to fix it.
It quickly turned out that although the program was not able to handle the load, the CPU was not very busy. The busiest core was only using about 30% CPU. So I suspected the thread contention problems in FastMM, switched it for SapMM and indeed - the problem went away. CPU load went up and application could again handle the load.
Crisis averted, I took time to find the real problem - excessive get/freemem calls in this program. As far as I could tell, there existed no tool to find that so in a true DIY manner I created my own ;)
