Eric's Home Page

Note: Retesting has shown that there are limits in the 2.2 kernel's ability to take advantage of multiple network cards, due to the fact that the network driver architecture was single-threaded, and that this was the primary cause of poor performance. Nevertheless, I stand by my basic conclusions: that the Mindcraft benchmark was a poorly crafted hack job, fundamentally flawed, and that the PC Labs retesting which detailed the basic problems was exactly what we in the Linux community needed, rather than Mindcraft's pitiful excuses and slanderous remarks.
--- Eric Lee Green

Mindcraft Reality Check

E.L. Green

I. Executive Summary:

Recently Mindcraft released a study comparing network performance of Windows NT 4.0 and Red Hat 5.2. The results of such performance benchmarks are of great interest to us and to the Linux community as a whole. Past benchmarks of Linux performance vs. its competitors have allowed us to perform many enhancements and improvements to the performance of the Linux kernel and associated programs, such as improved network performance (a result of benchmarks against FreeBSD), improved thread switch performance (a result of benchmarks against Windows NT), and improved performance of the kernel open() call (a result of benchmarks against Windows NT).

Fair comparisons are welcomed by those of us in the Linux community. Unfortunately, this Mindcraft comparison appears to be fatally flawed by misconfigurations of Linux, and thus is not useful for comparing the true performance of Red Hat 5.2 versus Windows NT.

II. Personnel Problems with Study

Mindcraft has proven themselves in the past to be masters of expertly tuning Windows NT. See, for example, their comparison testing of Windows NT versus Novell Netware 5.0, which showed Mindcraft's skill at expertly tuning Windows NT to its best. In this study, Mindcraft continues to display their expertise in tuning Windows NT. Unfortunately, they also demonstrate their lack of expertise in tuning Linux.

Tuning Linux, and any Unix in general, requires a certain level of expertise. This level of expertise cannot be gained within the three-day time period during which this study took place. This expertise can be gained via coursework from companies such as Learning Tree International or Eklektix, via concerted querying of the USENET newsgroups via DejaNews over a period of months, and by other such mechanisms over a period of months. This level of expertise is similar to the level of expertise needed to tune Windows NT, which generally requires MCSE coursework in order to fully master.

A useful comparison test between Windows NT and Red Hat Linux 5.2 would require that a Linux engineer with similar expertise in tuning high-end systems be on hand in order to properly tune the Linux system. When ZD Labs benchmarked Samba versus NT, they invited Jeremy Allison of the Samba team to assist in setting up their Samba configuration. A comparison test of properly tuned Linux and Windows NT systems would be quite constructive.

Unfortunately, from looking at the large number of technical mistakes (listed below) made while configuring the Linux system, it is quite obvious that such Linux expertise was not on hand. This violation of basic testing protocol means that the results of the test are not useful for valid comparison purposes.

III. Technical Problems with Study

A. Hardware Problems with Study

1) RAID controller Problems: The AMI RAID controller used in the study is known to have difficulties running under Linux. Linux Hardware Solutions was approached several months ago by AMI regarding using their RAID controllers on our systems, and after thorough evaluation, we concluded that the drivers for those controllers were still of beta-quality and not appropriate for a production server. AMI themselves to a certain extent admits this -- the version of the AMI RAID driver used by Mindcraft was version 0.92, i.e., it wasn't even mature enough to be awarded a 1.0 version number.

A fair test would use a RAID controller equally well supported by both Linux and Windows NT, such as the ICP-Vortex GDT line of RAID controllers used by Linux Hardware Solutions, or the Mylex line of RAID controllers used by VA Research Linux Solutions. In reality, a company buying a server of this size for use with Linux is going to buy one optimized for Linux with an appropriate RAID controller, rather than buy an off-the-shelf DELL server with a poorly supported RAID controller.

2) Disk Configuration Problems with Study: Both Windows NT and Linux were installed simultaneously on the tested system. The drives were partitioned such that Windows NT occupied half the drive, and Linux occupied the other half. Unfortunately, it is not stated within the study whether Linux was on the inner half or outer half of the drive. Note that the outermost tracks of a hard drive have approximately 1.5 to two times the data transfer rate of the innermost tracks, due to the larger number of sectors spinning past the head in a second's time.

B. Linux Configuration Problems with Study

1) Failure to configure Linux with large memory support A quick search of DejaNews with the parameters "2 gigabyte memory linux" swiftly turned up a message on how to configure Linux to access up to 2 gigabytes of physical memory. In their test, Mindcraft had Linux accessing only 96% of the memory that they had allocated for NT. This would be an immediate 4% performance hit in most file serving situations.

2) Failure to tune buffer allocation parameters Like all Unix-type operating systems, the Linux operating system has various tuning parameters to control the allocation of buffer memory, number of file handles available, and number of inode entries allowed. The default Linux 2.2 kernel will use up to 60% of physical memory as disk buffer cache, will allow up to 4096 open files, and will allow up to 12288 inodes. This is optimal for applications serving, but is completely inappropriate for file or web serving on a 1gb machine. Tuning the file buffer size so that more than 60% of memory can be used (90% in this example) can be accomplished by issuing the following command:

echo "2 10 90" >/proc/sys/vm/buffermem"

This is documented in the file /usr/src/linux/Documentation/sysctl/vm.txt along with many other tuning parameters, such as the 'bdflush' parameter. Jeremy Allison of the Samba team claims a factor of two improvement in Samba performance when these parameters are set properly.

Please note that queries of the DejaNews archives looking for queries by MindCraft researchers regarding tuning the Linux 2.2 kernel during the relevant time period turned up only one possible (but not probable) message, which asked a question about network card tuning. During the time span of January 1, 1999 through April 12, 1999, there were no messages posted to the Usenet asking about memory or buffer tuning for large memory machines.

3) Use of inappropriate kernel version The Linux 2.2.2 kernel's network stack had severe performance problems interacting with Microsoft clients. For more information, see the Kernel Traffic newsletter for March 4, 1999. Given the known problems with the 2.2.2 kernel, either the 2.2.1 kernel should have been used, or the 2.2.3 kernel should have been used.

4) Allocation of Swap not Specified Linux requires a swap partition for best operation. If the swap space is allocated on multiple drives (one swap partition per drive), the Linux VM system will utilize the drives in parallel, effectively using them as a RAID0 (striping) array. Unfortunately, we don't know how swap space was allocated in the Mindcraft test, and thus cannot duplicate their configuration.

C. Samba Configuration Problems

1) Recompilation of Samba with inappropriate settings: Red Hat Software has the Samba 2.0.x server available precompiled on their web site as a pre-packaged upgrade to go along with the 2.2 kernel. For some reason Mindcraft chose to use their own custom compiled package which was not compiled with the optimizations that Red Hat compiles into their own packages. In particular, the Mindcraft package was recompiled with the -O option rather than the -O2 option that Red Hat uses. In other words, the compiled binary was compiled with options that would generally make it noticably slower than the one shipped by Red Hat Software. To quote the GNU "gcc" documentation regarding the "-O2" option:

As compared to `-O', this option increases both compilation time and the performance of the generated code.

For a fair test, either the Red Hat provided Samba should have been used, or if a 'performance tweak' was needed to insure that both systems were optimized, the "egcs" C compiler (a standard part of Red Hat Linux) should have been used with the -mpentiumpro option to optimize the resulting code to take advantage of Pentium II features.

2) Inappropropriate use of the widelinks=no option: Andrew Tridgell of the Samba team reports that setting "widelinks=no" option will noticably slow performance of the Samba server. The "widelinks=no" option is a security feature intended for mixed-use systems, rather than for dedicated server use. For dedicated server use it is unnecessary, since Windows clients are unable to set filesystem links on a Samba server (only interactive Linux users can do so).

Please note that Jeremy Allison of the Samba team reports no queries on the Samba mailing lists or at the Samba home page that could have been about tuning Samba for use on large file servers during the time period in question.

D. Apache Configuration Problems

1) Inappropriate compilation:Again, for some unknown reason Mindcraft chose to re-compile Apache rather than use the default Red Hat version. This makes it difficult to compare "out of box" performance. If the purpose was to improve performance, using "egcs" to recompile it with the -mpentiumpro optimizations would have accomplished much more. Some Apache experts claim as much as 30% improvement by compiling Apache with the Pentium Pro optimizations.

2) Inappropriate settings for StartServers and SpareServers: If you are running an enterprise server expecting 150 or more simultaneous connections, you would set StartServers to 150 or greater. It's not as if there is a shortage of memory, after all.

3) Inadequate information to duplicate the test:Access to copies of the httpd configuration files is needed to further duplicate the test and diagnose the problems found there. For example, the "collapse" of the Apache server described therein is counter to what every other performance survey of Apache has found, which is that Apache reaches a maximum and then mildly degrades as overload continues.

A search of the DejaNews archives shows that a "will@whistlingfish.net" posted a query about this "collapse" problem. Unfortunately, it does not give enough information to make any sort of diagnosis possible (probably why it recieved no meaningful responses).

III. Non-Technical Problems with Study

The overall tone of the report was rather unfortunate. For example:

We posted notices on various Linux or Apache newsgroups and received no relevant responses

This implies that they made multiple requests for help ("notices") and that the Linux community, in violation of its reputation for excellence in technical support, dropped the ball. In actuality, exhaustive searching of the DejaNews archives by a team of Linux users has identified one single message as a possible candidate for these "notices". It was cross-posted under a different title between a Linux newsgroup and the Apache newsgroup, and recieved a reply which basically requested more information (albeit in a rather impatient manner). If the requested information was ever supplied, it was not ever posted to the newsgroup where other members of the Linux community could add their own comments.

The report is full of similar statements which could be interpreted as showing partiality or being inappropriate. Given the fact that this study was funded by Microsoft Corporation, Mindcraft should have taken special care to protect their reputation by appearing as impartial as possible.

IV. Conclusions

We welcome the opportunity to work with Mindcraft to resolve these glaring issues and produce a quality analysis. The Linux community is extremely interested in a fair, impartial analysis of the performance of Linux as compared with competitors. Such analysis has been very valuable in the past, and promises to be equally valuable to the technical advancement of Linux in the future.

Unfortunately, given the severe technical shortcomings of this report, we must respectfully request that Mindcraft withdraw it until such issues are resolved. A report which is perceived to be biased is not in the best interests of either Mindcraft or its client, and may in fact be embarrasing to both.

Credits: Thanks to the readers and editors of Linux Today and Linux Weekly News for proofreading and technical assistance.

Mandatory disclaimers: The contents of this page represent the opinions of Eric Lee Green and other members of the Linux community.

Created with PHP 4. Last modified Fri, 06 Dec 2002 10:27:39 -0500.

Menu: