With macOS 10.13, otherwise known as High Sierra, Apple introduced an ambitious EFI update. Several of the changes include: the introduction of the Apple File System, support for NVMe drives, and the usual batch of security updates. However, these EFI updates can cause some unwanted behavior when you test your Apple computer’s memory. At Beetstech, we use a long-time industry standard, MemTest86 to perform a comprehensive test of each computer’s RAM.

But never the type to blindly accept test results, strange testing outcomes led us to discover a bug in MemTest86 affecting computers running the new EFI firmware. In short, the newly updated EFI causes MemTest86 to incorrectly fail certain tests. But there is good news: while normal operation of MemTest86 is limited under these new EFI updates, we also discovered some simple workarounds for testing your Apple’s memory in MemTest86.

So let’s dive into how we discovered the MemTest bug, devised a reliable work-around, and get into some nitty gritty details of MemTest86 operation.

MemTest86: A program that provides robust testing of your computer's memory.

MemTest86

MemTest86 is a program that tests the entire range of a computer system’s RAM. MemTest86 is made and maintained by PassMark Software. The software is installed onto a USB flash drive or CD. Boot into the device on which you installed MemTest86 and the software will run the tests automatically.

Configuring MemTest86

There are up to fourteen different tests that MemTest86 can do on your computer’s memory. The first 10 tests – tests number 0 through 9 – are straightforward: each test writes a specific pattern of bits to a section of your computer’s memory. Then the test goes through that section of memory and reads the values back to see if they are correct before moving to the next section of your computer’s memory and repeating the process until the test is complete. Tests number 11 and 12 also exhibit this behavior, but are more tailored to specific processor architectures (i.e. 64-bit instructions, 128-bit SIMD instructions). The tests in which we are particularly interested are tests number 10 and 13.

The goal of test 10 is to see if your memory modules can hold their value over a span of time. For each section of memory that is currently being tested, MemTest86 performs two steps. MemTest86 will write all zeros to the section of memory, wait five minutes, and then read all the addresses to see if any of the data changed. The second step is mostly the same as the first step, except that MemTest86 writes all ones instead of all zeros. A single pass of this test can take several minutes to several hours to complete depending on how much memory is in your computer.

Test 13 is generally much faster than test 10. The goal of test 13 is to determine if your memory modules have a defect that is present in memory modules released after 2010. This test writes a random pattern of bits to a range of memory addresses and then repeatedly reads two memory addresses. This repeated reading action can cause some of the tested memory addresses to change in value if your memory is defective.

The Issue

It is these latter two tests that do not work well with Apple’s recent EFI updates. We discovered this issue when we had a batch of five 13” MacBook Airs and two 15” MacBook Pros – all from 2015 – that were all running MemTest86 simultaneously. All of the MacBook Airs were running macOS 10.12, one MacBook Pro was running macOS 10.13, and the other MacBook Pro was running OS X 10.11. Two of the MacBook Airs passed the tests with no issues, while the other three Airs and both the Pros would fail spectacularly. Each failing computer would throw at least 1000 errors, all of which occurred in a single pass of test 10. Sometimes a computer would fail test 10 multiple times, each time resulting in about 1000 errors. The MacBook Pros would also fail test 13 in such a manner that could cause the test to throw over 100,000 errors.

The MacBook Pro with OS X 10.11 is a computer that we’ve had for almost two years, and our internal documentation noted that this particular MacBook Pro had passed MemTest86 previously. We felt that the fact that five of these computers would fail in the same manner every time was not merely a coincidence of these five having bad memory; the results were too spectacular. We determined that there might be an issue with MemTest86 as well.

Browsing through the MemTest86 support forums revealed an interesting fact to us about the software. While the software is robust in its testing, the correctness of the tests can be sometimes at the mercy of factors beyond the condition of the tested memory. For example, a stick of memory that is tested in one computer could fail some tests, but then that same stick could pass all tests when it is tested in another computer. A potential cause for false errors could be from the firmware interface of the computer system; an example of such a firmware interface would be the EFI system present on recent Apple computers.

We know that with the release with macOS 10.13 came an important EFI update which allows the previously mentioned Apple File System and NVMe compatibility. We also know that the EFI update that comes with macOS 10.13 is more permanent than an installation of the operating system; regressing to an older operating system, such as macOS 10.12, will not remove the EFI update. When we double checked the EFI versions of the computers in question, all the failing computers had an EFI version that came with or came after the macOS 10.13 EFI update. The two computers that were passing the tests had EFI versions that came before the macOS 10.13 update.

A BootROM version that is problematic with MemTest86. These can identified with the 'B00' suffix.

The Experiment and Its Results

With the two MacBook Airs that successfully complete MemTest86 and our suspicions about the EFI update, we decided to perform an experiment in two phases. In the first phase of the test, we would have the two MacBook Airs undergo an extended run MemTest86. The goal of this phase was  to establish that these two have no memory errors. In the second phase, we would then install macOS 10.13 on one of the two MacBook Airs and have both run MemTest86 normally to observe the results. The goal of the second phase was to confirm our suspicions that recent EFI updates for Apple computers would interfere with the performance of MemTest86.

The results from the first phase of the test were promising. We set the two computers to perform twenty passes of MemTest86 and let them run overnight. When we came back in the morning, we observed that neither of the machines had reported any errors during the testing. We took the results of this phase to mean that neither of the two MacBook Airs had faulty memory and proceeded to the next phase.

The results from the second phase were also promising in that the results confirmed our suspicions: the newer EFI updates will interfere with the ability for MemTest86 software to test an Apple computer’s memory. The MacBook Air that we had updated to macOS 10.13 reported ten thousand errors while the other Air reported no errors. While we cannot say for certain what aspect of the EFI update is throwing these errors, we can say with confidence that updating to the newer versions of the EFI will causes errors with MemTest86 when running the tests normally.

A Description of the Failures

Of course with these results came the next question: how can we reliably test an Apple computer’s memory in such a way that we can trust the results? To do this, we studied how these computers with the newer EFI versions would fail in MemTest86. While the affected computers wouldn’t fail every pass of MemTest86, there was a very high chance that they would fail a pass at some point. It just so happened that when these computers did fail MemTest86, these computers would fail in a consistent manner for both tests 10 and 13.

In the case of test 10 failing, the problematic address ranges were always the same. The failures always occur in the memory address range of 0x00 to 0x1000. In particular, the grand failure is heralded by a few failures that occur in the first stage of this test. When MemTest86 writes all zeros to the section under test, the addresses of 0x00 and 0x04 tend to fail.  When they do fail, we observe that each address fails in the same way (e.g. address 0x00 will always read 0x02000112 and address 0x04 will always read 0x0800000 when they fail this part of the test). In the second, ‘write all ones’ pass of test 10, every single address in the range fails. Of note, all the values of the data in each address are all zeros, as though MemTest86 was unable to write any data to this memory address.

Two errors in the first pattern are followed by 1024 errors in the second pattern.

In the case of test 13 failing, the address ranges also appear to be between 0x00 and 0x1000. Much like how the latter portions of test 10 did not appear to write any data to the affected address ranges, test 13 also appears to not write any data to its memory addresses; the expected input is randomized, but the data from the read is all zeroes. There is also an issue with the firmware locking out access to all the processor cores except core zero for this test, though what effect this has on the performance of the test is yet to be determined.

Test 13 in the process of failing. This test is likely to generate several thousand more errors.

The Workaround

In order to test the memory of an Apple computer that has or has had High Sierra installed, we must let MemTest86 run twice. One of the key features of MemTest86 is that you can select the range of memory on which to perform the tests. Knowing this, we devised a solution for testing your Apple computer’s memory that works with EFI version that both precede and supersede the EFI update that comes with macOS 10.13.

When we boot into MemTest86, we will want to go to the configuration menu. Usually the tests will begin automatically, but we can bypass that by pressing any key after the program is loaded. In the configuration menu, there is a menu called ‘Address Range.’ Select that menu and you will see three options: Lower Limit, Upper Limit, and Reset limits to default. Adjusting these values is how we will be testing the range of memory in chunks. Note that the upper limit will be different from computer to computer and reflects how much memory your computer has.

We will be using hexadecimal values when adjusting memory addresses for the tests, which requires us to provide a prefix of 0x before we provide a value. Have caution that if you press xanywhere in the MemTest86 configuration menu that you will exit the program! But if you pressxwhen adjusting the memory address range values, you should be fine.

Setting the lower address limit to 0x1000

For the first pass, set the lower limit option to ‘0x1000’ and let the upper limit remain the default value. Run the test. Once the test is finished, you can be certain that any errors reported are valid, though hopefully no such errors occur! For the second pass, go back to the address menu, set the lower limit to ‘0x00’, and set the upper limit to ‘0x1000’. Run the test again and observe the results.

Resetting the lower limit to 0x0 and changing the upper limit to 0x1000

This method allows all the tests to perform over the entire range of memory within the Apple computer. We have yet to encounter any false errors when using this test method, though we invite everyone to try this out and let us know how it works for them.

Going forward

We’ve informed PassMark of our findings and they are well aware of this issue with MemTest86 and recent Apple EFI updates. While PassMark has provided validation for our new testing method, future iterations of MemTest86 are going to treat Apple computers differently. PassMark has stated that the newer versions of MemTest86 will start blacklisting certain memory addresses from Apple computers to prevent the appearance of these false errors.

Apple has been made aware of the issue, but to date Apple has yet to respond to any reports of this issue. Given Apple’s recent lobbying efforts, it almost seems more than coincidence that updating your Apple computer to macOS 10.13 affects the performance of tests 10 and 13. Here at Beetstech, we will continue testing thoroughly every aspect of the Apple computers we receive, including all the memory addresses that we can test. With macOS Mojave 10.14 coming out on the 24th of this month, we anticipate finding any issues that come with the new release.

Useful Links

• MemTest86 download