Things Learned from Performance Tuning
February 10th, 2008
One major blocker for a software project to finish is unsatisfactory performance. Engineers always are asked to tune the performance to customers’ satisfaction no matter what. Performance tuning itself is a very interesting topic and I’d like to share some interesting things that I’ve encountered before.
Two main performance tuning targets are reducing execution time or shortening waiting time. Reducing execution time is typically related to back-end applications. I had involved in a banking software project which had a 19.5-hour batch. Well, you know this is way too absurd to be acceptable. After continuous tuning for a month, the batch later took 19 minutes to complete. It might not be common to see such drastic results, however, I did learn something very interesting:
- The table design of a database is very important. Most of the (banking) projects that I’ve involved before design their database based on existing table structures from the customers. Unfortunately, theses table structures are not likely well optimized (at least for the project you are working on). In the very beginning of the project, we need to identify the most frequent or critical query path, and design the table structures accordingly.
- The golden rule of store procedure selection is to make sure the first selected data set is the smaller one. When a cursor points to a smaller data set, the loop count will be fewer and performance is typically better. The example I gave above improved from 19.5 hours to 6 only by switching the order of two nested select statement.
- As a team leader, you need to have real assessment for your team members’ capabilities. Some people might consider this as micro-management, however, I found it a super common scenario. One good way to have an idea about that is to talk to your team members and see what he/she plans to deal with the issues. If there’s no plan, or the execution/executability of the plan is questionable, you’ll need a plan B.
- Customers and PMs usually are trapped in the myth of scalability: they demand things that have far more capacity than needed, or are super flexible that can perform all kinds of ridiculous magic. It’s important to bring them down to the earth and get the job done.
- Most frustrations come from the issue of “real requirements”. In many cases, the performance issue is just a product of miserable management, poor communications, political complexity, or the combination of these craps. For example, if two VPs of your customer’s company ask for technically conflicting features, what will you do? One way is to wait another five years for Moore’s Law to defeat them by raw computation power. Sometimes we just need little help from very higher ups to get the deal settled.
- Many people learned the idea that improvement of algorithm is more efficient than optimizing the code. However, this idea is based on the assumption that the original algorithm is implemented with quality code. Please make sure the code is quality code before wasting time to figure out better faster fancier algorithms. Lame code can screw the best algorithm on the planet without any problem.
- If you work with a gigantic team, it is almost fated that the product of your team will be slow due to the nature of bureaucracy. It’s purely a management problem and need courage and luck to overcome.
Shortening waiting time typically is not related to technical issues, unless the design is bad from the very beginning, or the customer’s network infrastructure is not capable. Most solutions involve psychological effects, or perceptual performance per se. For example, I’ve encountered that my customer complains our report generating program is too slow, because it takes 5 minutes to generate a 50-page report. Okay. Later I give them another version, with a dialog showing a progress bar and flashing text regarding to the progress of report generation, and the customer is very happy about it. Guess what? The new program takes 7 minutes to generate the same 50-page report 🙂 Well, the trick is very simple: user feel that things are “moving” if they see a responsive UI instead of a stalled one.
By the way, in my previous article about the myth of memory usage, we see another form of perceptual performance although it is not related to shortening waiting time. Tuning the perceptual performance could be tricky and need a lot of work in usability researches. If you do not have a usability guy covering your back, it’s time to have more chats with the customers and try to figure out what they think.