Hi everyone, we are seeing a few cases where a System Center 2012 R2 Configuration Manager update scan is failing so I wanted to mention it here in case any of you happened to run across it. Typically the scenario is that a ConfigMgr 2012 R2 client is requesting an update scan but the Windows Update Agent on 32-bit Windows 7 computers fails to return the scan results to Configuration Manager. This causes the Configuration Manager client to report incorrect compliance status as a result. It is further evident that the updates fail to install on the Windows 7 32-bit clients when ConfigMgr requests the update cycle, however if you use the Windows Update control panel applet the updates will usually install.
You will also notice a message similar to the following in WindowsUpdate.log:
WARNING: ISusInternal::GetUpdateMetadata2 failed, hr=8007000E
At its core this is a memory allocation issue, thus 64-bit Windows 7 computers will not see this error because the address space is effectively unlimited. They will, however, exhibit high memory and high CPU usage, possibly affecting performance. Note that x86 clients will also exhibit high memory usage (around 1.2-1.4GB).
We have also had reports of this affecting System Center Endpoint Protection updates deployed by Configuration Manager as well as a few reports of Windows 8 x86 clients being affected.
A hotfix for the Windows Update Agent is currently in development. The update will change how the update metadata is loaded into memory. Pending the results of final testing, this fix should be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.
Windows x86 clients have a much smaller addressable memory range than 64-bit clients. With enough updates, the data returned to Configuration Manager is large enough that RPC can’t allocate the shared section used to return the data. This is because overall memory usage of the svchost.exe process hosting the Windows Update Agent process is extremely high.
The result is that there is either not enough memory available to satisfy the request or there is not a large enough contiguous block of free memory to satisfy the request.
1. Move wuauserv (Windows Update Agent) to its own SVCHost.exe instance by completing the following:
NOTE Always backup the WSUS database (SUSDB) prior to performing any changes such as those described here.
After declining unneeded updates, re-index the susdb, then run the WSUS Server Cleanup Wizard to remove unneeded updates as appropriate. Note that this will remove the updates from any Configuration Manager software update groups of which it is part.
3. Set user VA to 3072 MB. This is done by running bcdedit /set IncreaseUserVA 3072at a Command Prompt on the client. This will free up another 1GB of memory in the user address space. Note that this requires a restart of the client computer.
Please be aware that when attempting to address this problem it may be necessary to use all of the steps above. Option 1 should be strongly considered because wuauserv (the Windows Update agent service) is in a shared svchost process and stopping wuauserv does not unload the svchost process. If the failures are due to memory fragmentation, the memory of the svchost process will remain fragmented because wuauserv is reloaded into that instance.
If there are too many updates in WSUS, the script may fail to get the updates and time out. The exact number of updates will vary greatly depending on many environmental variables (e.g. the number of operating systems present, the number of versions of Office deployed, the number of Internet Explorer versions, etc.). In the event that the script times out, you will have to resort to manually declining updates from the WSUS console. Instructions to manually decline updates are included in the section titled Cleaning up WSUS from the WSUS console below.
NOTE The default WSUS server port is 80, but if you have installed WSUS to a custom IIS site it is probably using a different port. You will need to determine what port WSUS is using and change the –Portparameter in the examples below to that port.
IMPORTANT Make sure to take a backup of the susdb before declining the updates!
The argument –DeclineLastLevelOnly declines only those updates that do notsupersede any other update. If you omit this argument, any update that is superseded will be declined, leaving only updates that are not superseded in a state other than ‘declined.’
First, run the script with the -SkipDeclineswitch to see how many superseded updates are in WSUS. For example, to do a test run against WSUS Server without SSL you would use the following command:
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -SkipDecline
Next, you can decline only the updates that are superseded but do not supersede updates (leaf-level updates):
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -DeclineLastLevelOnly
This next command will decline allsuperseded updates:
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -UseSSL -Port 8080
1. Open the Windows Update Services MMC.
2. Select the All Updates View : Set the display to show the Approval status of 'Any except Declined' with a Status of 'Any'. Click Refresh.
3. Display the Supersedence Column : Right-click the column headers and select Supersedence.
4. Sort by Supersedence : Left-click on the Supersede Column.
5. Select and decline the superseded updates.
The updates to be declined have one of two particular flowchart symbols for their updates as shown below. Select the correct updates and decline them by either right-clicking the selected updates and clicking Decline, or by pressing the Decline button in the action pane.
IMPORTANT You must select only the updates that have one of the two icons below. There are a total of 3 icons that may appear here and you are to select the following 2 only.
- This means the update is superseded by another update and it supersedes another update.
- This means this update is superseded by another update. Below is a snapshot that shows updates that have been superseded by other updates.
As mentioned earlier, a hotfix for this issue is currently in development and is scheduled to be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.
You will also notice a message similar to the following in WindowsUpdate.log:
WARNING: ISusInternal::GetUpdateMetadata2 failed, hr=8007000E
At its core this is a memory allocation issue, thus 64-bit Windows 7 computers will not see this error because the address space is effectively unlimited. They will, however, exhibit high memory and high CPU usage, possibly affecting performance. Note that x86 clients will also exhibit high memory usage (around 1.2-1.4GB).
We have also had reports of this affecting System Center Endpoint Protection updates deployed by Configuration Manager as well as a few reports of Windows 8 x86 clients being affected.
A hotfix for the Windows Update Agent is currently in development. The update will change how the update metadata is loaded into memory. Pending the results of final testing, this fix should be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.
Root Cause
Because Configuration Manager does compliance reporting, it has extremely broad scan criteria to determine what is applicable. This is different from a typical scan performed by the Windows Update agent which requires a smaller subset of criteria. If an update is on WSUS and in a non-declined state, the Configuration Manager scan criteria will cause the Windows Update agent to try to evaluate it.Windows x86 clients have a much smaller addressable memory range than 64-bit clients. With enough updates, the data returned to Configuration Manager is large enough that RPC can’t allocate the shared section used to return the data. This is because overall memory usage of the svchost.exe process hosting the Windows Update Agent process is extremely high.
The result is that there is either not enough memory available to satisfy the request or there is not a large enough contiguous block of free memory to satisfy the request.
Workarounds
Please note that the workarounds listed below may not resolve the issue in every environment, thus It’s possible that some machines or applications will continue to have problems even after these workarounds have been implemented. It is important to test these workarounds in a testing or lab environment before deploying into production.1. Move wuauserv (Windows Update Agent) to its own SVCHost.exe instance by completing the following:
a. On the client, open a Command Prompt and run sc config wuauserv type= own2. Decline any unnecessary updates on the WSUS server. This may avoid the problem because declined updates do not get offered to clients during scans. Unneeded updates include superseded updates, updates for products and/or classifications that are not present in the client environment, as well as expired updates. You can manually decline the updates within the WSUS console or use the script documented in the Cleaning up WSUS with a Scriptsection below.
b. Stop and then start wuauserv.
NOTE Always backup the WSUS database (SUSDB) prior to performing any changes such as those described here.
After declining unneeded updates, re-index the susdb, then run the WSUS Server Cleanup Wizard to remove unneeded updates as appropriate. Note that this will remove the updates from any Configuration Manager software update groups of which it is part.
3. Set user VA to 3072 MB. This is done by running bcdedit /set IncreaseUserVA 3072at a Command Prompt on the client. This will free up another 1GB of memory in the user address space. Note that this requires a restart of the client computer.
Please be aware that when attempting to address this problem it may be necessary to use all of the steps above. Option 1 should be strongly considered because wuauserv (the Windows Update agent service) is in a shared svchost process and stopping wuauserv does not unload the svchost process. If the failures are due to memory fragmentation, the memory of the svchost process will remain fragmented because wuauserv is reloaded into that instance.
Cleaning up WSUS with a Script
We have provided an example of a script that can be used to clean up WSUS that will allow scripted declining of superseded updates in your WSUS environment. Updates need to be declined at the top-level WSUS instance and replicated to downstream WSUS instances that are configured for replica mode. You will need to run the script on any WSUS instance running in Autonomous mode. To use the script you must rename it to Decline-SupersededUpdates.ps1. As always, it is important to test this script in a lab environment before deploying into production.If there are too many updates in WSUS, the script may fail to get the updates and time out. The exact number of updates will vary greatly depending on many environmental variables (e.g. the number of operating systems present, the number of versions of Office deployed, the number of Internet Explorer versions, etc.). In the event that the script times out, you will have to resort to manually declining updates from the WSUS console. Instructions to manually decline updates are included in the section titled Cleaning up WSUS from the WSUS console below.
NOTE The default WSUS server port is 80, but if you have installed WSUS to a custom IIS site it is probably using a different port. You will need to determine what port WSUS is using and change the –Portparameter in the examples below to that port.
IMPORTANT Make sure to take a backup of the susdb before declining the updates!
The argument –DeclineLastLevelOnly declines only those updates that do notsupersede any other update. If you omit this argument, any update that is superseded will be declined, leaving only updates that are not superseded in a state other than ‘declined.’
First, run the script with the -SkipDeclineswitch to see how many superseded updates are in WSUS. For example, to do a test run against WSUS Server without SSL you would use the following command:
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -SkipDecline
Next, you can decline only the updates that are superseded but do not supersede updates (leaf-level updates):
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -DeclineLastLevelOnly
This next command will decline allsuperseded updates:
Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -UseSSL -Port 8080
Cleaning up WSUS from the WSUS console
If you have to (or want to) decline updates manually, the WSUS console has an icon that will tell you if an update is superseded.1. Open the Windows Update Services MMC.
2. Select the All Updates View : Set the display to show the Approval status of 'Any except Declined' with a Status of 'Any'. Click Refresh.
3. Display the Supersedence Column : Right-click the column headers and select Supersedence.
4. Sort by Supersedence : Left-click on the Supersede Column.
5. Select and decline the superseded updates.
The updates to be declined have one of two particular flowchart symbols for their updates as shown below. Select the correct updates and decline them by either right-clicking the selected updates and clicking Decline, or by pressing the Decline button in the action pane.
IMPORTANT You must select only the updates that have one of the two icons below. There are a total of 3 icons that may appear here and you are to select the following 2 only.
- This means the update is superseded by another update and it supersedes another update.
- This means this update is superseded by another update. Below is a snapshot that shows updates that have been superseded by other updates.
As mentioned earlier, a hotfix for this issue is currently in development and is scheduled to be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.
No comments:
Post a Comment