Understanding 4M Page Size Extensions
on the Pentium Processor

by

Robert R. Collins


It's been more than three years since Intel first published the Pentium Family User's Manual. The Manual omitted discussion of some new, advanced programming features. Intel originally planned to release this information in its manuals, but instead, put this information in a document commonly referred to as "Appendix H" (formally known as the Supplement to the Pentium Processor User's Manual ) and required recipients to sign a 15-year nondisclosure agreement (NDA). This decision has been the focus of a controversy concerning Intel's right to protect its intellectual property versus the rights of all programmers to have access to information that will benefit their programs. Another point of contention is the NDA itself. Intel claims that anybody needing this information will never be denied it, as long as they sign the NDA. But several stories have circulated regarding programmers being denied because Intel claims they don't need the information. This has spawned a community of programmers dedicated to reverse engineering these features and publishing their findings on Internet news groups and the World Wide Web. But is all of this necessary?

Intel has promised that the not-yet-released Pentium Pro Processor Family Developer's Manual will contain information on many of these advanced features, perhaps even a description of 4-MB paging.

Four-MB paging allows the operating system to access very large data structures without constantly referencing the Translation Lookaside Buffer (TLB), which is used by the processor to cache virtual-to-physical address translations for the most recently used pages of memory. This feature is most useful to operating-system developers who want a single page of memory dedicated to the OS kernel or a large data structure, such as a video-frame buffer. Information about 4-MB paging has been publicly documented by Intel - but you need to know where to look to find it, In order to get a complete description of Pentium's 4-MB pages, you need to read both the Pentium Family User's Manual, Volume 3 (P/N 241430) and the i860TM XP Microprocessor Data Book (P/N 240874).

In the Pentium manuals, there are at least nine references to 4-MB pages. This is a good start to reverse engineering 4-MB pages. These references give you the necessary clues to write software that unlocks the secrets of page-size extensions (PSE). However, such an effort is unnecessary. The Intel i860 XP processor documentation claims the i860 XP is page-level compatible with the Intel 386, Intel 486, and Pentium processors. This compatibility is noteworthy because the i860 XP also supports 4-MB pages, and its documentation provides a complete description of the 4-MB paging mechanism (see i860TM XP Microprocessor Data Book, section 2.4). All that's needed to obtain an Appendix H description of 4-MB pages are a few references from the Pentium manuals and the description of 4-MB pages from the i860 XP manual.

A 4-KB Page Backgrounder

When paging is enabled, linear addresses program-visible addresses) are mapped to hysical addresses (bus addresses). Paging makes it possible to execute programs much larger than the computer's available amount of memory. When the microprocessor needs more memory, it generates a page fault to demand that a portion of memory be mapped between the hard disk and main memory. Memory is partitioned into contiguous blocks, called "page frames." Each page frame is 4 KB. The Pentium paging mechanism consists of the following:

The PDBR is CR3, and points to the base of the page directory. Each page-directory entry (PDE) points to the page tables for 4 MB of memory. The PDE contains control information and the pointers to the page tables. Like the PDE, each page-table entry (PTE) contains control information, but points to a 4-KB page frame. Linear addresses are converted to physical addresses by using a 20-bit pointer in a page table and combining it with the low-order 12 bits of the linear address to form a 32-bit physical address. For purposes of conversion, the linear address is broken into three parts:

The upper 20 bits of the PTE are then combined with the low-order 12 bits of the linear address to form the physical address. There is a direct relationship between the sizes of these three fields and the page size. The lower 12 bits can address 212, or 4 KB of memory. Hence, each PTE controls 4 KB of memory. The amount of memory controlled by each PDE is determined by the number of address bits used as an index into the page table, plus the number of bits used as the page- frame index. The PTE index is 10 bits, and the page-frame index is 12 bits, making 222, or 4 MB of memory controlled by each PDE. This association will be important in understanding the 4-MB paging mechanism. Figure 1 shows how linear addresses are translated to physical addresses for 4-KB pages.

Figure 1 -- Page Translation for 4KB Page Sizes

Page Translation 4 MB Mechanism

Making the jump to 4MB pages

With an understanding of the 4-KB paging mechanism, it's not difficult to deduce the 4-MB paging mechanism. Recall that each page-directory entry controls 4 MB of memory. Now imagine how Figure 1 would look if the page-table lookup were eliminated. The page-frame index would increase from 12 bits to 22 bits, thus allowing direct control of a 4-MB page size. The 20-bit pointer in the page directory would be reduced to a 10-bit pointer, pointing directly to the 4-MB page frame of memory. With the page-table lookup eliminated, the page directory points directly to a 4-MB page frame. This describes how 4-MB pages are implemented in the i860 XP (i860™ XP Microprocessor Data Book, section 2.4). But the question remains: Are 4-MB i860 XP pages compatible with 4-MB Pentium pages? To answer that question, we need to compare the i860 and Pentium manuals.

The i860 manual claims that the i860 4-KB paging mechanism is compatible with the x86 implementation. A comparison of page-directory format and page-table format substantiates this claim. The page-size (PS) bit of the i860 page directory shares the same location as the Pentium's PS bit (see i860™ XP Microprocessor Data Book, Figure 2.13). With this information, you can assume they are compatible, and look more closely at the Pentium manual for the mechanics of enabling and using 4-MB pages.

Volume 3 of the Pentium manual describes how CR4.PSE enables PSE's and 4-MB pages, but refers you to Appendix H for more information. Later in the Pentium manual, bit 7 of the PDE is identified as the PS bit. Without CR4.PSE=1, the Pentium will always use Intel 486-compatible (4-KB) paging, regardless of the setting of the PDE.PS bit. Similarly, when CR4.PSE=1, and PDE.PS=0, Pentium still uses Intel 486-compatible 4-KB pages. But when CR4.PSE=1, and PDE.PS=1, Pentium uses an i860 XP-compatible 4-MB paging translation.

The linear address for a 4-MB page is converted to a physical address in much the same manner as 4-KB pages. However, the access to the page table is omitted. The high-order 10 bits form an index into the page directory. The page directory no longer contains a 20-bit pointer to a page table, but instead contains a 10-bit pointer to the 4-MB page frame of memory. This convention mandates that all 4-MB pages reside on 4-MB boundaries. The 10-bit pointer in the page directory then is combined with the low-order 22 bits of the linear address to form the 32-bit physical address.

Figure 2 describes the 4-MB and 4-KB paging translation mechanism. Ironically, Figure 11-16 in Pentium Processor Family Developer's Manual, Volume 3, 1993 edition, contained a virtually identical picture. Intel obviously recognized the significance of this pictorial representation of 4-MB pages. Subsequent editions of the Pentium manual were substantially modified to remove the visual representation of the 4-MB paging mechanism.

Figure 2-- Page Translation for 4MB and 4KB Page Sizes

Page Translation for 4 MB and 4 KB Page Sizes

Side-effects and Caveats of 4-MB Pages

There are side-effects and caveats to enabling 4-MB pages. Consider the following excerpt from the Pentium Processor Family Developer's Manual, Volume 3, section 23.2.14.1, which discusses compatibility with previous Intel processors:

In other words, if any reserved bit in the PDE or PTE is 1, a page fault will occur. This does not occur when CR4.PSE=0, but does when PSE's are enabled (CR4.PSE=1).[1] Every bit in CR4 enables a behavioral extension to the Intel 486 processor. In essence, CR4 bits enable/disable incompatibilities with the Intel 486. Therefore, it is a natural extension of enabling 4-MB pages to enable more rigorous type checking of the PDE and PTE. Unfortunately, even then, the aforementioned reference isn't completely accurate. Setting some reserved bits does generate an exception, while setting others does not. This behavior contradicts the Intel documentation. If the Pentium was originally intended to behave as documented, then the documentation didn't get modified to accurately reflect the correct behavior when relaxed type checking for reserved bits was implemented. Table 1 shows all of the Pentium paging structures. All positions in the PDE and PTE marked as reserved will generate a page-fault exception when CR4.PSE=1. All positions in CR3, the PDE, and PTE marked as "0" are reserved, but don't generate a page fault when CR4.PSE=1. Table 2 describes the meaning of all of the fields listed in Table 1.

[*] It might be tempting to believe that the "page-directory pointer" is another name for the CR3 register. This assumption would be incorrect. Actually, the mention of the page-directory pointer is a mistake. This refers to a paging structure for a new paging feature that was to be implemented in the Pentium. This new paging feature was allegedly implemented in beta silicon, but removed before production, and now appears in the Pentium Pro. I'll discuss this in my next column.

The Intel documentation also doesn't tell the whole story of the error code generated by page faults. When CR4.PSE=1, and a 1 is detected in a reserved-bit position of the PDE or PTE, the page-fault error code indicates that an attempt was made to set a reserved bit in a paging structure. This indication is reflected in bit 3 of the page-fault error code. If set to 1, then an attempt was made to set a reserved bit in the PDE or PTE. In Figure 14-7 of the Pentium Processor Family Developer's Manual, Volume 3, 1993 edition, this behavior was correctly documented, but it was removed in subsequent editions. Table 3 shows an accurate representation of the page-fault error code, as shown in the 1993 edition of the Pentium manual.

Table 1 -- Structures used in Pentium paging translations

Structures used in Pentium paging translations


Table 2 -- Descriptions of paging extension fields

Descriptions of paging extension fields


Table 3 -- Page fault error code

Page fault error code

TLB Translation

According to the 1995 edition of the Pentium user's manual, the Pentium has one code TLB and two data TLBs (Pentium Processor Family Developer's Manual, Volume 1, 1995 edition, section 33.2.1.2). The data TLBs consist of a 64-entry TLB for 4-KB page translations, and an 8-entry TLB for 4-MB page translation. The code TLB is a single 32- entry TLB which is shared by 4-KB and 4-MB page translations. The 4-MB code pages are cached in multiples of 4 KB. When the Pentium caches a 4-MB code page in the TLB, it initially uses only a single TLB entry. A code access beyond the initial 4 KB of memory associated with this TLB accesses the PDE as if it were a 4-KB page, and is given its own TLB entry.

TLB Invalidation

You'd assume that enabling and disabling 4-MB pages (CR4.PSE) would invalidate the TLB, as writing to CR3 does. However, this does not occur. A potentially dangerous situation arises when a user wants to disable 4-MB pages when a 4-MB page is still cached in the TLB. Suppose the PDEs were modified with a different paging translation and point to a different area of physical memory than the 4-MB pages (this would be natural to assume, as it complies with the whole purpose of paging). Once CR4.PSE is cleared, then any 4-MB TLB entries still cached remain in effect until they are evicted or until the TLB is invalidated. (Once CR4.PSE=0, TLB entries for 4-MB data pages will never get evicted, since they have their own dedicated TLB.) Any subsequent memory (or code) accesses while the old 4-MB TLB still is cached would retrieve incorrect data. Therefore, before 4-MB paging can be disabled, all 4-MB PDEs must be modified back to 4-KB PDEs. Once the PDEs are modified, CR4.PSE can be cleared, or the TLB invalidated (which effectively disables 4-MB paging). Some could consider this a bug, but Intel's documentation states that it's the operating-system writer's responsibility to manage the paging mechanism, including in-validating the TLB (Pentium Processor Family Developer's 3fanual, Volume 3, section 11.3.5).

Testing the Hypothesis

Now that we have an understanding of 4-MB paging, it should be easy to write characterization code that confirms our hypothesis. To detect whether or not 4-MB pages are implemented in Pentium as they are in the i860 XP, you could follow these steps:

The key to this technique is to read from one location in memory if 4-MB pages work or another location if they don't (so you don't page fault). This approach is demonstrated in 4MPAGES.ASM.

What to Try Next

You could write more characterization code to prove whether or not any other functional extensions are enabled by setting CR4.PSE. The listings available electronically demonstrate the page-faulting behavior of PSE. I've also included a program that detects the TLB size and associativity. Finally, another program demonstrates that writing any values to CR4.PSE will not invalidate the TLB.


Source code examples

The following examples are available for viewing and download.

View source code for 4MPAGES.ASM:
ftp://ftp.x86.org/source/4mpages/4mpages.asm

Download source code and executable archive:
ftp://ftp.x86.org/dloads/4MPAGES.ZIP


Endnotes:

  1. Beta copies of the Pentium documentation correctly described this behavior. However, as the documentation was put into production, the mention that this behavior depended upon CR4.PSE=1 was removed. The wording between the two documents was identical, except for the omission of this one sentence. Naturally, one could assume this omission was intentional to obscure the true behavior of the Pentium Processor from Intel's competitors and anybody else who tried to reverse-engineer this feature.

Back to Dr. Dobb's Undocumented Corner home page