http://insomniacgeek.com/how-to-install-moss-on-windows-server-2008-r2/
First a disclaimer. Microsoft doesn’t support this scenario, so you will be on your own.
When you try to install MOSS 2007 on Windows Server 2008 R2, you will be prompted by this error message:
and you are stuck.
The only way to install MOSS 2007 on Windows Server 2008 R2 is to slipstream MOSS Service Pack 2 into the installation media. I will now run through the steps to do so.
Prerequisites:
MOSS 2007 installation media. (Either the .iso file from MSDN, or the DVD)
WSS 3.0 Service Pack 2 (Remember to get the x64 version, since Windows 2008 R2 is 64bit only)
MOSS 2007 Service Pack 2 (Remember to get the x64 version, since Windows 2008 R2 is 64bit only)
First, extract the contents of your MOSS 2007 media .iso file (or simply copy the files if you are using the DVD) to a temporary installation folder. In my case this will be H:\MOSS Installation Media. There is an TechNet article on how to slipstream updates into MOSS, but I will go through the steps here also.
Open a command prompt and change directory to where you have the service pack files. Extract the contents to the Updates folder of your installation folder.
First the WSS Service Pack files:
officeserver2007sp2-kb953334-x64-fullfile-en-us.exe /extract:"H:\MOSS Installation Media\x64\Updates"
Then the MOSS 2007 service pack files:
officeserver2007sp2-kb953334-x64-fullfile-en-us.exe /extract:"H:\MOSS Installation Media\x64\Updates"
Now, you will need to delete the file wsssetup.dll from the Updated folder, otherwise it will only install the WSS SP2.
You can now install WSS 3.0 and MOSS 2007 SP2 using set setup.cmd found in the installation folder.
Friday, January 21, 2011
Installing Exchange 2010 SP1
http://blogs.pointbridge.com/Blogs/olson_andy/Pages/Post.aspx?_ID=5
I have had several customers ask me about Exchange 2010 SP1 and whether I have had any issues with the installation. In response to several such requests, I thought I'd share my experience installing Exchange 2010 SP1 in my test lab. In this article, my lab consists of a single server running the following:
Server 2008 R2 Enterprise
Multi-Role Exchange 2010 Server (HUB, CAS, MB)
I ran into issues immediately when the installation Readiness Checks returned the following hotfixes must be installed before SP1 can be installed:
Hub Transport Role Prerequisites:
Office 2010 Filter Packs (http://go.microsoft.com/fwlink/?LinkID=191548)
Client Access Role Prerequisites:
Hotfix for KB982867
Hotfix for KB979744
Hotfix for KB983440
Hotfix for KB977020
Mailbox Role Prerequisites:
Office 2010 Filter Packs (http://go.microsoft.com/fwlink/?LinkID=191548)
Screen shots of the errors I received follow:
Hub Transport Role Prerequisites:
Client Access Role Prerequisites:
Mailbox Role Prerequisites:
The Exchange Team Blog has some information about the hotfix prerequisites required for Exchange 2010 SP1 that you can read about here: http://msexchangeteam.com/archive/2010/09/01/456094.aspx. Hoping that I could avoid downloading each hotfix independently, I tried running Windows Update, making sure to check for application updates, but that did not install any of the required hotfixes. I also tried to install Exchange 2010 Roll Up 4 hoping that the required hotfixes might be included in Roll Up 4, but that did not contain any of the required hotfixes either.
Here is an excerpt from the Exchange Team Blog that helps explain why the hotfixes are necessary:
"Given that the Exchange team released SP1 earlier than what was planned and announced earlier, it did not align with some of the work with the Windows platform. As a result, some hotfixes are available from MSDN/Connect, and some require that you request them online using the links in the corresponding KBAs."
Reading the above excerpt did not give me a whole lot of comfort because it seemed to indicate that finding and downloading all of the hotfixes was not going to be exceptionally straight forward. The link provided in the Hub Transport Rule Prerequisites took me exactly where I needed to go. I found it interesting that I needed to install the Office 2010 Filter Packs again, because I know that I had installed them once already during my initial installation. However, reinstalling the Office 2010 Filter Packs presented me with no issues. I then downloaded all 4 remaining hotfixes separately and installed them one at a time.
Here are the links to the downloads that I required:
Hub Transport Role Prerequisites:
Office 2010 Filter Packs
http://go.microsoft.com/fwlink/?LinkID=191548
Client Access Role Prerequisites:
Hotfix for KB982867
http://code.msdn.microsoft.com/KB982867/Release/ProjectReleases.aspx?ReleaseId=4520
Hotfix for KB979744
http://code.msdn.microsoft.com/KB979744/Release/ProjectReleases.aspx?ReleaseId=3993
Hotfix for KB983440
https://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=29092
Hotfix for KB977020
http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=27977
Mailbox Role Prerequisites:
Office 2010 Filter Packs
http://go.microsoft.com/fwlink/?LinkID=191548 (same hotfix required for the Hub Transport Role)
As the Exchange Team Blog site mentioned, the download experience for each hotfix was slightly different than you might expect. Make sure to download the correct version of each file. In my case I downloaded the Windows6.1 hotfixes with an x64.msu extension. For example:
Two of the hotfixes that I installed required a reboot. However, I installed all 4 hotfixes along with reinstalling the Office 2010 Filter Packs one after the other, and then only rebooted once after all 5 were installed. After I installed all hotfixes, installing Exchange 2010 SP1 worked just fine. Note that SP1 also requires a reboot after installation, so plan to reboot each of your production Exchange 2010 Servers at least twice to complete the SP1 installation.
The Exchange Team Blog mentions some additional issues that some administrators have experienced after installing Exchange 2010 SP1. I have not experienced any of those issues as of yet. So far the only challenge I have had with Exchange 2010 SP1 has been the installation. However, once I found, downloaded, and installed all the prerequisites my installation proceeded without issue.
I have had several customers ask me about Exchange 2010 SP1 and whether I have had any issues with the installation. In response to several such requests, I thought I'd share my experience installing Exchange 2010 SP1 in my test lab. In this article, my lab consists of a single server running the following:
Server 2008 R2 Enterprise
Multi-Role Exchange 2010 Server (HUB, CAS, MB)
I ran into issues immediately when the installation Readiness Checks returned the following hotfixes must be installed before SP1 can be installed:
Hub Transport Role Prerequisites:
Office 2010 Filter Packs (http://go.microsoft.com/fwlink/?LinkID=191548)
Client Access Role Prerequisites:
Hotfix for KB982867
Hotfix for KB979744
Hotfix for KB983440
Hotfix for KB977020
Mailbox Role Prerequisites:
Office 2010 Filter Packs (http://go.microsoft.com/fwlink/?LinkID=191548)
Screen shots of the errors I received follow:
Hub Transport Role Prerequisites:
Client Access Role Prerequisites:
Mailbox Role Prerequisites:
The Exchange Team Blog has some information about the hotfix prerequisites required for Exchange 2010 SP1 that you can read about here: http://msexchangeteam.com/archive/2010/09/01/456094.aspx. Hoping that I could avoid downloading each hotfix independently, I tried running Windows Update, making sure to check for application updates, but that did not install any of the required hotfixes. I also tried to install Exchange 2010 Roll Up 4 hoping that the required hotfixes might be included in Roll Up 4, but that did not contain any of the required hotfixes either.
Here is an excerpt from the Exchange Team Blog that helps explain why the hotfixes are necessary:
"Given that the Exchange team released SP1 earlier than what was planned and announced earlier, it did not align with some of the work with the Windows platform. As a result, some hotfixes are available from MSDN/Connect, and some require that you request them online using the links in the corresponding KBAs."
Reading the above excerpt did not give me a whole lot of comfort because it seemed to indicate that finding and downloading all of the hotfixes was not going to be exceptionally straight forward. The link provided in the Hub Transport Rule Prerequisites took me exactly where I needed to go. I found it interesting that I needed to install the Office 2010 Filter Packs again, because I know that I had installed them once already during my initial installation. However, reinstalling the Office 2010 Filter Packs presented me with no issues. I then downloaded all 4 remaining hotfixes separately and installed them one at a time.
Here are the links to the downloads that I required:
Hub Transport Role Prerequisites:
Office 2010 Filter Packs
http://go.microsoft.com/fwlink/?LinkID=191548
Client Access Role Prerequisites:
Hotfix for KB982867
http://code.msdn.microsoft.com/KB982867/Release/ProjectReleases.aspx?ReleaseId=4520
Hotfix for KB979744
http://code.msdn.microsoft.com/KB979744/Release/ProjectReleases.aspx?ReleaseId=3993
Hotfix for KB983440
https://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=29092
Hotfix for KB977020
http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=27977
Mailbox Role Prerequisites:
Office 2010 Filter Packs
http://go.microsoft.com/fwlink/?LinkID=191548 (same hotfix required for the Hub Transport Role)
As the Exchange Team Blog site mentioned, the download experience for each hotfix was slightly different than you might expect. Make sure to download the correct version of each file. In my case I downloaded the Windows6.1 hotfixes with an x64.msu extension. For example:
Two of the hotfixes that I installed required a reboot. However, I installed all 4 hotfixes along with reinstalling the Office 2010 Filter Packs one after the other, and then only rebooted once after all 5 were installed. After I installed all hotfixes, installing Exchange 2010 SP1 worked just fine. Note that SP1 also requires a reboot after installation, so plan to reboot each of your production Exchange 2010 Servers at least twice to complete the SP1 installation.
The Exchange Team Blog mentions some additional issues that some administrators have experienced after installing Exchange 2010 SP1. I have not experienced any of those issues as of yet. So far the only challenge I have had with Exchange 2010 SP1 has been the installation. However, once I found, downloaded, and installed all the prerequisites my installation proceeded without issue.
Wednesday, November 24, 2010
How to use ADPlus utility to troubleshoot issues related to process hang and process crash.
http://www.symantec.com/connect/articles/how-use-adplus-utility-troubleshoot-issues-related-process-hang-and-process-crash
ADPlus is a tool from Microsoft Product Support Services (PSS) that can troubleshoot any process or application that stops responding (hangs) or fails (crashes). These tool can be used if you have any issues related to ccApp.exe crashing or smc service encountered a problem and need to close.
Steps to follow before using the tool.
1.Download Debugging tools from Microsoft website and install the same on the problem machine.
2.There is no need to download any symbols for debugging only the tool is sufficient.
Steps to follow in order to generate the logs for process crash.
1.Create a folder on any of the drive to collect the logs.
2.Open command prompt and browse to the location where the debugging tool is installed (i.e.:\Program Files\Debugging Tools for Windows (x86))
3.Execute the command adplus.vbs -crash -pn processname -o "path to output folder"
4.Please refer to the below mentioned example.
C:\Program Files\Debugging Tools for Windows (x86)>adplus.vbs -crash -pn SmcGui.exe -o "D:\output\"
Steps to follow in order to generate the logs for process hang.
1.Create a folder on any of the drive to collect the logs.
2.Open command prompt and browse to the location where the debugging tool is installed (i.e.:\Program Files\Debugging Tools for Windows (x86))
3.Execute the command adplus.vbs -hang -pn processname -o "path to output folder"
4.Please refer to the below mentioned example.
C:\Program Files\Debugging Tools for Windows (x86)>adplus.vbs -crash -pn SmcGui.exe -o "D:\output\"
The above mentioned command will associate the debugger with the process and at the point when they crash it will generate the logs in the output folder which you can transfer to support for further analysis.
For reference please refer to the below mentioned Microsoft Knowledge base article
http://support.microsoft.com/kb/286350
ADPlus is a tool from Microsoft Product Support Services (PSS) that can troubleshoot any process or application that stops responding (hangs) or fails (crashes). These tool can be used if you have any issues related to ccApp.exe crashing or smc service encountered a problem and need to close.
Steps to follow before using the tool.
1.Download Debugging tools from Microsoft website and install the same on the problem machine.
2.There is no need to download any symbols for debugging only the tool is sufficient.
Steps to follow in order to generate the logs for process crash.
1.Create a folder on any of the drive to collect the logs.
2.Open command prompt and browse to the location where the debugging tool is installed (i.e.
3.Execute the command adplus.vbs -crash -pn processname -o "path to output folder"
4.Please refer to the below mentioned example.
C:\Program Files\Debugging Tools for Windows (x86)>adplus.vbs -crash -pn SmcGui.exe -o "D:\output\"
Steps to follow in order to generate the logs for process hang.
1.Create a folder on any of the drive to collect the logs.
2.Open command prompt and browse to the location where the debugging tool is installed (i.e.
3.Execute the command adplus.vbs -hang -pn processname -o "path to output folder"
4.Please refer to the below mentioned example.
C:\Program Files\Debugging Tools for Windows (x86)>adplus.vbs -crash -pn SmcGui.exe -o "D:\output\"
The above mentioned command will associate the debugger with the process and at the point when they crash it will generate the logs in the output folder which you can transfer to support for further analysis.
For reference please refer to the below mentioned Microsoft Knowledge base article
http://support.microsoft.com/kb/286350
Friday, September 3, 2010
SQL 2008 Clustering & Video
SQL 2008 Video
http://www.mssqltips.com/webcastlist.asp?m1
Regarding your question, you need to setup Windows 2008 R2 cluster first before installing SQL 2008 clustering. Win2008 R2 cluster setup is very similar as Win2008, please refer to the following article for details steps
indows Server 2008 Clustering Documents, Whitepapers, Webcasts, Groups
http://blogs.msdn.com/b/clustering/archive/2008/05/10/8483427.aspx
Also, for SQL 2008 cluster, you need install it on each node which is different from SQL 2005 cluster. Please refer to the following article for details
How to: Create a New SQL Server Failover Cluster (Setup)
http://msdn.microsoft.com/en-us/library/ms179530.aspx
Actually, there are step by step guide with screenshots from the following links
http://www.mssqltips.com/tip.asp?tip=1687
http://www.mssqltips.com/tip.asp?tip=1698
http://www.mssqltips.com/tip.asp?tip=1709
http://www.mssqltips.com/tip.asp?tip=1721
http://www.mssqltips.com/webcastlist.asp?m1
Regarding your question, you need to setup Windows 2008 R2 cluster first before installing SQL 2008 clustering. Win2008 R2 cluster setup is very similar as Win2008, please refer to the following article for details steps
indows Server 2008 Clustering Documents, Whitepapers, Webcasts, Groups
http://blogs.msdn.com/b/clustering/archive/2008/05/10/8483427.aspx
Also, for SQL 2008 cluster, you need install it on each node which is different from SQL 2005 cluster. Please refer to the following article for details
How to: Create a New SQL Server Failover Cluster (Setup)
http://msdn.microsoft.com/en-us/library/ms179530.aspx
Actually, there are step by step guide with screenshots from the following links
http://www.mssqltips.com/tip.asp?tip=1687
http://www.mssqltips.com/tip.asp?tip=1698
http://www.mssqltips.com/tip.asp?tip=1709
http://www.mssqltips.com/tip.asp?tip=1721
Monday, August 2, 2010
Sharepoint Forum
SharePoint
http://social.msdn.microsoft.com/Forums/en-US/category/sharepoint
Windows 2008 Clustering
http://social.technet.microsoft.com/Forums/en-US/winserverClustering/threads
http://social.msdn.microsoft.com/Forums/en-US/category/sharepoint
Windows 2008 Clustering
http://social.technet.microsoft.com/Forums/en-US/winserverClustering/threads
Tuesday, June 22, 2010
Paged and Non-paged Pool Issues on Exchange 2000/2003
http://blogs.technet.com/b/dblanch/archive/2009/04/18/paged-and-non-paged-pool-issues-on-exchange-2000-2003.aspx
This blog provides some guidance on how to optimize Exchange 2000/2003 on Windows OS for paged and non-paged pool issues.
Symptoms
Event log might show
Event ID 2020
Event Type: Error
Event Source: Srv
Event ID: 2020
Description: The server was unable to allocate from the system paged pool because the pool was empty.
Event ID 2019
Event Type: Error
Event Source: Srv
Event ID: 2019
Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.Other symptoms of pool exhaustion on the system include application or process hangs, out of resource errors reported by drivers or applications, the server becomes slow or refuses additional requests and connections, or all of the above!
Understanding Pool
When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool. Pool memory is allocated statically during Windows startup. Available pool memory depends on several factors to include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.
Pool memory is not the amount of RAM on the system. It is a segment of the virtual memory or address space that Windows reserves on boot. These pools are finite because x86 OS can only address 2^32==4GB. By default, Windows uses 2GB for applications and 2GB for kernel.
These pools are used by either the kernel directly, indirectly by its support of various structures due to application requests on the system (CreateFile for example), or drivers installed on the system for their memory allocations made via the kernel pool allocation functions.
NonPaged means that this memory when allocated will not be paged to disk and thus resident at all times, which is an important feature for drivers. Paged conversely can be paged out to disk. In the end though, all this memory is allocated through a common set of functions, most common is ExAllocatePoolWithTag.
System PTEs
Page Tables are built for each process address space. The Page Table maps logical virtual addresses for a process to physical memory locations. System Page Table Entries (PTEs) are used to map system pages such as I/O space, Kernel stacks, and memory descriptor lists. Tuning the memory by using the USERVA switch in conjunction with the /3GB switch can often stave off PTE depletion issues.
/USERVA
A boot.ini switch used for more precise tuning of user and kernel virtual memory space in the Windows Server 2003. Use this switch in conjunction with /3GB switch in the Boot.ini file to tune reduce the User-mode space, allowing the difference to be returned to Kernel mode. Standard configuration is to use /USERVA=3030. However there might be several situations where you need to allow for more PTEs to become available. On an Exchange server, the value of USERVA should not be lower than 2970 without consulting Windows Performance team.
/3GB
A boot.ini switch that allocates 1 GB to the kernel and 3 GB to the User-mode space. Using this switch reduces the memory available for Nonpaged Pool, Paged Pool, & System Page Table Entries (PTEs).
/PAE
A Boot.ini switch. When more than 4GB of physical memory is used on the system, the process of paging memory to the disk increases dramatically, and performance may be negatively impacted. The Windows memory managers use PAE to provide more physical memory to the operating system. This reduces the need to swap the memory in and out of the page file and results in increased performance. All the memory management and allocation of the PAE memory is handled by the memory manager independently of the running programs. It is now a best practice to allow the PAE kernel to load on an Exchange 2003 server.
Pool Tags
A pool tag is a four-byte character that is associated with a dynamically allocated chunk of pool memory. The tag is specified by a driver when it allocates the memory. The routine ExAllocatePoolWithTag is called to allocate pool memory. Pool tags are useful for identifying which drivers are allocating nonpaged and paged pool memory.
By the Numbers
Review the chart at the bottom of this document to identify the default boot up of Paged Pool, NonPaged Pool, and Free System PTE as it applies to your system configuration.
A typical large-scale Exchange 2003 server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention. Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.
In general, a system should always have around 10,000 free System PTE’s. If the value gets below 5,000, then the system could hang temporarily.
64-bit resolves this, right?
Not quite. While 64-bit programs can take advantage of the 16-TB tuning model (8 terabytes User and 8 terabytes Kernel) that 64-bit OS has, 32-bit programs still use the 4-GB tuning model (2 GB User and 2 GB Kernel). This means that 32-bit processes that run on 64-bit versions of Windows run in a 4-GB tuning model. So by running 32-bit programs on 64-bit OS, you could still run into limitations.
Troubleshooting Memory Depletion
Here are some basic troubleshooting steps to identifying the problem with Pool Memory depletion on Exchange 2000/2003 server.
1.Is the system showing any symptoms of pool memory depletion (event log, hangs, etc)?
2.Check system performance using Task Manager and performance monitor (Memory Counters: Available Mbytes, Free System Page Table Entries, Pool Nonpaged Bytes, & Pool Paged Bytes)
3.Use Poolmon to capture the system’s memory allocations of paged and non-paged kernel pools. See the section below on using Poolmon. (poolmon –n –b –g)
4.Review the poolmon output and identify any tags that are consuming a high amount of Paged or NonPaged memory. Compare the output to your baseline or another system.
5.To search for a driver that uses a specific tag, use the findstr /m /l /s xxxx *.sys command from the cmd prompt (where xxxx is the tag name).
6.If you suspect a tag or driver, research that driver (KB, Internet, etc) for any known issues and latest hotfixes.
7.If needed, review the pool information using debugging tools (see debugging section below)
Common Causes of Pool Depletion
•Problem with a driver or application taking more than needed (leak)
•With Exchange 2000/2003, having more than 4 GB of physical RAM (Each byte of physical RAM that is installed in a server requires some kernel memory to address and manage it. The more RAM that is installed, the more kernel address space must be reserved for it. Address space may be borrowed from paged pool memory to satisfy this demand.)
•Having a Public Folder store mounted on a mailbox server that has many mailboxes. If mailboxes are connecting to the PF store, the additional connections require additional system resource to be consumed.
•User token size (http://support.microsoft.com/kb/912376)
NOTE: Size of the PF store does not matter – it is the connections to the store
Data Collection
If experiencing symptoms, collect the following data:
•From a remote location, take a performance monitor capture of the system
•Capture EXBPA, MPSReports, and Poolmon data from the server
•If needed, capture a user dump of the system (see debugging section below)
Common Best Practices
The following are some best practices for managing paged and nonpaged pool issues on Exchange 2000/2003.
NOTE: Before performing these steps, run Poolmon on the server at difference intervals to keep track of system changes
•Run Exchange Best Practice Analyzer (EXBPA) and resolve all of the issues that it identifies. EXBPA identifies many of the common configuration settings for improving memory.
•Check for proper Boot.ini configuration (/3GB, /USERVA, /basevideo, /PAE, etc)
•Within the System BIOS, disable Hot-Add memory -and- set the DynamicMemory registry key to a value of 1 (http://technet.microsoft.com/en-us/library/aa996104.aspx)
•Disable Windows Scalable Networking Pack and TCP Chimney on the server
•Disable Checksum Offloading by setting the DisableTaskOffload regkey to a value of 1 (http://support.microsoft.com/kb/904946)
•Check the Network card settings and make sure that “Number of Receive Descriptors” is set to default (NIC Properties > Advanced)
•Disable or remove unwanted Applications or drivers from the system
•Distribute heavy-connection users evenly across multiple servers. Heavy-connection users are likely to be those who have multiple computers or devices and those who are mobile users. This includes mobile, telephony, archiving, and mail client connections.
•A Public Folder store can impact paged and nonpaged memory. Consider using a dedicated PF server for large environments.
•Identify user TOKEN (TOKE) size (http://support.microsoft.com/kb/912376). If this is high, refer to KB 912376 to reduce token size or distribute users with large tokens across multiple servers.
•Apply relevant hotfixes & updated drivers to the system
•Restrict unauthorized clients and applications from accessing Exchange server. This can be done by isolating the subnet between clients and servers and by setting the Outlook connection range value on the server (also see KB 288894)
•Using terminal services in application server mode on an Exchange 2000/2003 server can drastically impact performance on the server. Do not use terminal services in application mode!
•Reduce File Cache Lifetime to 5 minutes (300 decimal) (http://support.microsoft.com/kb/267551)
•Set POP3/IMAP Protocol Logging to 0 (http://support.microsoft.com/kb/299778)
•Implement MaxPercentPoolThreads and AdditionalPoolThreadsPerProc (http://technet.microsoft.com/en-us/library/aa997700.aspx)
•Move the EXIFS free list (Flst) and auxiliary free list (AuxL) to paged pool memory
◦HKLM\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters
•Set “AuxFreeListInPagedPool” = REG_DWORD 0x00000001
•Set “FreeListInPagedPool” = REG_DWORD 0x00000001
•Set EnableAggressiveMemoryUsage for IIS to 1 (http://support.microsoft.com/kb/820129)
NOTE: There are other settings and adjustments that can be made to the system to potentially improve Paged and NonPaged Pool performance. Those adjustments should only be made after completing the above steps and consulting with Microsoft Services.
Using Poolmon
Poolmon is Windows Support Tool that displays data that the operating system collects about memory allocations from the system paged and non-paged kernel pools, and the memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag.
NOTE: Pool tagging is permanently enabled on Windows Server 2003. However, in earlier OS versions (including XP), you must use Gflags.exe to enable pool tagging.
1. Open a command prompt and view the available options: Poolmon /?
-s Display session pool
-n [Logfile] Take a pool snapshot (Logfile maybe specified, default is poolsnap.log)
-c [LocalTagFile] Display driver information using LocalTagFile
-g [PoolTagFile] Display driver information using PoolTagFile
-itag Include the tag
-xtag Exclude the tag
-e Display totals
-t Sort by tags
-a Sort by allocs
-u | -b Sort by Bytes
-f Sort by free
-d Sort by diff
-m Sort by each
-l Highlight
-p First turns on nonpaged, second turns on paged
-( | -) Increase parenthesis
-r Print memory summary information
2. Copy the Pooltag.txt file from the Windows Debugging tools directory to the system running poolmon. This will allow you to map the driver to the tag.
3. Next run Poolmon -n –b -g
Review the poolsnap.log. The tags should be sorted by byte usage.
References
•Pool Tags List - http://alter.org.ua/en/docs/win/pooltag/index1.php
•Introduction to Pool Tags - http://blogs.technet.com/askperf/archive/2008/04/11/an-introduction-to-pool-tags.aspx
•Memory Management - Understanding Pool Resources - http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
•Who's Using the Pool? - http://www.microsoft.com/whdc/driver/tips/PoolMem.mspx
•Understanding Pool Consumption - http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx
•How to find pool tags that are used by third-party drivers - http://support.microsoft.com/kb/298102
•Exchange Server 2003 Processor and Memory Scalability - http://technet.microsoft.com/en-us/library/aa996184.aspx
•CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server - http://support.microsoft.com/kb/827281
•How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server - http://support.microsoft.com/kb/912376
•Ruling Out Memory-Bound Problems - http://technet.microsoft.com/en-us/library/aa998384.aspx
•KB 177415 How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
•Poolmon on Technet - http://technet.microsoft.com/pt-pt/library/cc737099.aspx
•http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx
•http://blogs.msdn.com/ntdebugging/archive/2008/03/26/nonpagedpool-depletion.aspx
•http://blogs.msdn.com/ntdebugging/archive/2008/05/08/tracking-down-mmst-paged-pool-usage.aspx
This blog provides some guidance on how to optimize Exchange 2000/2003 on Windows OS for paged and non-paged pool issues.
Symptoms
Event log might show
Event ID 2020
Event Type: Error
Event Source: Srv
Event ID: 2020
Description: The server was unable to allocate from the system paged pool because the pool was empty.
Event ID 2019
Event Type: Error
Event Source: Srv
Event ID: 2019
Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.Other symptoms of pool exhaustion on the system include application or process hangs, out of resource errors reported by drivers or applications, the server becomes slow or refuses additional requests and connections, or all of the above!
Understanding Pool
When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool. Pool memory is allocated statically during Windows startup. Available pool memory depends on several factors to include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.
Pool memory is not the amount of RAM on the system. It is a segment of the virtual memory or address space that Windows reserves on boot. These pools are finite because x86 OS can only address 2^32==4GB. By default, Windows uses 2GB for applications and 2GB for kernel.
These pools are used by either the kernel directly, indirectly by its support of various structures due to application requests on the system (CreateFile for example), or drivers installed on the system for their memory allocations made via the kernel pool allocation functions.
NonPaged means that this memory when allocated will not be paged to disk and thus resident at all times, which is an important feature for drivers. Paged conversely can be paged out to disk. In the end though, all this memory is allocated through a common set of functions, most common is ExAllocatePoolWithTag.
System PTEs
Page Tables are built for each process address space. The Page Table maps logical virtual addresses for a process to physical memory locations. System Page Table Entries (PTEs) are used to map system pages such as I/O space, Kernel stacks, and memory descriptor lists. Tuning the memory by using the USERVA switch in conjunction with the /3GB switch can often stave off PTE depletion issues.
/USERVA
A boot.ini switch used for more precise tuning of user and kernel virtual memory space in the Windows Server 2003. Use this switch in conjunction with /3GB switch in the Boot.ini file to tune reduce the User-mode space, allowing the difference to be returned to Kernel mode. Standard configuration is to use /USERVA=3030. However there might be several situations where you need to allow for more PTEs to become available. On an Exchange server, the value of USERVA should not be lower than 2970 without consulting Windows Performance team.
/3GB
A boot.ini switch that allocates 1 GB to the kernel and 3 GB to the User-mode space. Using this switch reduces the memory available for Nonpaged Pool, Paged Pool, & System Page Table Entries (PTEs).
/PAE
A Boot.ini switch. When more than 4GB of physical memory is used on the system, the process of paging memory to the disk increases dramatically, and performance may be negatively impacted. The Windows memory managers use PAE to provide more physical memory to the operating system. This reduces the need to swap the memory in and out of the page file and results in increased performance. All the memory management and allocation of the PAE memory is handled by the memory manager independently of the running programs. It is now a best practice to allow the PAE kernel to load on an Exchange 2003 server.
Pool Tags
A pool tag is a four-byte character that is associated with a dynamically allocated chunk of pool memory. The tag is specified by a driver when it allocates the memory. The routine ExAllocatePoolWithTag is called to allocate pool memory. Pool tags are useful for identifying which drivers are allocating nonpaged and paged pool memory.
By the Numbers
Review the chart at the bottom of this document to identify the default boot up of Paged Pool, NonPaged Pool, and Free System PTE as it applies to your system configuration.
A typical large-scale Exchange 2003 server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention. Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.
In general, a system should always have around 10,000 free System PTE’s. If the value gets below 5,000, then the system could hang temporarily.
64-bit resolves this, right?
Not quite. While 64-bit programs can take advantage of the 16-TB tuning model (8 terabytes User and 8 terabytes Kernel) that 64-bit OS has, 32-bit programs still use the 4-GB tuning model (2 GB User and 2 GB Kernel). This means that 32-bit processes that run on 64-bit versions of Windows run in a 4-GB tuning model. So by running 32-bit programs on 64-bit OS, you could still run into limitations.
Troubleshooting Memory Depletion
Here are some basic troubleshooting steps to identifying the problem with Pool Memory depletion on Exchange 2000/2003 server.
1.Is the system showing any symptoms of pool memory depletion (event log, hangs, etc)?
2.Check system performance using Task Manager and performance monitor (Memory Counters: Available Mbytes, Free System Page Table Entries, Pool Nonpaged Bytes, & Pool Paged Bytes)
3.Use Poolmon to capture the system’s memory allocations of paged and non-paged kernel pools. See the section below on using Poolmon. (poolmon –n –b –g)
4.Review the poolmon output and identify any tags that are consuming a high amount of Paged or NonPaged memory. Compare the output to your baseline or another system.
5.To search for a driver that uses a specific tag, use the findstr /m /l /s xxxx *.sys command from the cmd prompt (where xxxx is the tag name).
6.If you suspect a tag or driver, research that driver (KB, Internet, etc) for any known issues and latest hotfixes.
7.If needed, review the pool information using debugging tools (see debugging section below)
Common Causes of Pool Depletion
•Problem with a driver or application taking more than needed (leak)
•With Exchange 2000/2003, having more than 4 GB of physical RAM (Each byte of physical RAM that is installed in a server requires some kernel memory to address and manage it. The more RAM that is installed, the more kernel address space must be reserved for it. Address space may be borrowed from paged pool memory to satisfy this demand.)
•Having a Public Folder store mounted on a mailbox server that has many mailboxes. If mailboxes are connecting to the PF store, the additional connections require additional system resource to be consumed.
•User token size (http://support.microsoft.com/kb/912376)
NOTE: Size of the PF store does not matter – it is the connections to the store
Data Collection
If experiencing symptoms, collect the following data:
•From a remote location, take a performance monitor capture of the system
•Capture EXBPA, MPSReports, and Poolmon data from the server
•If needed, capture a user dump of the system (see debugging section below)
Common Best Practices
The following are some best practices for managing paged and nonpaged pool issues on Exchange 2000/2003.
NOTE: Before performing these steps, run Poolmon on the server at difference intervals to keep track of system changes
•Run Exchange Best Practice Analyzer (EXBPA) and resolve all of the issues that it identifies. EXBPA identifies many of the common configuration settings for improving memory.
•Check for proper Boot.ini configuration (/3GB, /USERVA, /basevideo, /PAE, etc)
•Within the System BIOS, disable Hot-Add memory -and- set the DynamicMemory registry key to a value of 1 (http://technet.microsoft.com/en-us/library/aa996104.aspx)
•Disable Windows Scalable Networking Pack and TCP Chimney on the server
•Disable Checksum Offloading by setting the DisableTaskOffload regkey to a value of 1 (http://support.microsoft.com/kb/904946)
•Check the Network card settings and make sure that “Number of Receive Descriptors” is set to default (NIC Properties > Advanced)
•Disable or remove unwanted Applications or drivers from the system
•Distribute heavy-connection users evenly across multiple servers. Heavy-connection users are likely to be those who have multiple computers or devices and those who are mobile users. This includes mobile, telephony, archiving, and mail client connections.
•A Public Folder store can impact paged and nonpaged memory. Consider using a dedicated PF server for large environments.
•Identify user TOKEN (TOKE) size (http://support.microsoft.com/kb/912376). If this is high, refer to KB 912376 to reduce token size or distribute users with large tokens across multiple servers.
•Apply relevant hotfixes & updated drivers to the system
•Restrict unauthorized clients and applications from accessing Exchange server. This can be done by isolating the subnet between clients and servers and by setting the Outlook connection range value on the server (also see KB 288894)
•Using terminal services in application server mode on an Exchange 2000/2003 server can drastically impact performance on the server. Do not use terminal services in application mode!
•Reduce File Cache Lifetime to 5 minutes (300 decimal) (http://support.microsoft.com/kb/267551)
•Set POP3/IMAP Protocol Logging to 0 (http://support.microsoft.com/kb/299778)
•Implement MaxPercentPoolThreads and AdditionalPoolThreadsPerProc (http://technet.microsoft.com/en-us/library/aa997700.aspx)
•Move the EXIFS free list (Flst) and auxiliary free list (AuxL) to paged pool memory
◦HKLM\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters
•Set “AuxFreeListInPagedPool” = REG_DWORD 0x00000001
•Set “FreeListInPagedPool” = REG_DWORD 0x00000001
•Set EnableAggressiveMemoryUsage for IIS to 1 (http://support.microsoft.com/kb/820129)
NOTE: There are other settings and adjustments that can be made to the system to potentially improve Paged and NonPaged Pool performance. Those adjustments should only be made after completing the above steps and consulting with Microsoft Services.
Using Poolmon
Poolmon is Windows Support Tool that displays data that the operating system collects about memory allocations from the system paged and non-paged kernel pools, and the memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag.
NOTE: Pool tagging is permanently enabled on Windows Server 2003. However, in earlier OS versions (including XP), you must use Gflags.exe to enable pool tagging.
1. Open a command prompt and view the available options: Poolmon /?
-s Display session pool
-n [Logfile] Take a pool snapshot (Logfile maybe specified, default is poolsnap.log)
-c [LocalTagFile] Display driver information using LocalTagFile
-g [PoolTagFile] Display driver information using PoolTagFile
-itag Include the tag
-xtag Exclude the tag
-e Display totals
-t Sort by tags
-a Sort by allocs
-u | -b Sort by Bytes
-f Sort by free
-d Sort by diff
-m Sort by each
-l Highlight
-p First turns on nonpaged, second turns on paged
-( | -) Increase parenthesis
-r Print memory summary information
2. Copy the Pooltag.txt file from the Windows Debugging tools directory to the system running poolmon. This will allow you to map the driver to the tag.
3. Next run Poolmon -n –b -g
Review the poolsnap.log. The tags should be sorted by byte usage.
References
•Pool Tags List - http://alter.org.ua/en/docs/win/pooltag/index1.php
•Introduction to Pool Tags - http://blogs.technet.com/askperf/archive/2008/04/11/an-introduction-to-pool-tags.aspx
•Memory Management - Understanding Pool Resources - http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
•Who's Using the Pool? - http://www.microsoft.com/whdc/driver/tips/PoolMem.mspx
•Understanding Pool Consumption - http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx
•How to find pool tags that are used by third-party drivers - http://support.microsoft.com/kb/298102
•Exchange Server 2003 Processor and Memory Scalability - http://technet.microsoft.com/en-us/library/aa996184.aspx
•CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server - http://support.microsoft.com/kb/827281
•How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server - http://support.microsoft.com/kb/912376
•Ruling Out Memory-Bound Problems - http://technet.microsoft.com/en-us/library/aa998384.aspx
•KB 177415 How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
•Poolmon on Technet - http://technet.microsoft.com/pt-pt/library/cc737099.aspx
•http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx
•http://blogs.msdn.com/ntdebugging/archive/2008/03/26/nonpagedpool-depletion.aspx
•http://blogs.msdn.com/ntdebugging/archive/2008/05/08/tracking-down-mmst-paged-pool-usage.aspx
Subscribe to:
Posts (Atom)