A replacement for SCCM Add-CMDeviceCollectionDirectMembershipRule PowerShell cmdlet

TL;DR – The native Add-CMDeviceCollectionDirectMembershipRule PowerShell cmdlet sucks for adding more than 100 devices, use this replacement script instead.

How fast is good enough? When is the default, too slow?

I guess most of us have been spoiled with modern machines: Quad Xeon Procesors, couple hundred GB of ram, NVME cache drives, and Petabytes of storage at our command.

And don’t get me started with modern database indexing, you want to know what the average annual rainfall on the Spanish Plains are? If I don’t get 2 million responses within a half a second, I’ll be surprised, My Fair Lady.

But sometimes as a developer we need to account for actual performance, we can’t just use the default process and expect it to work in all scenarios to scale.

Background

Been working on a ConfigMgr project in an environment with a machine count well over ~300,000 devices. And we were prototyping a project that involved creating Device Collections and adding computers to the Collections using Direct Membership Rules.

Our design phase was complete, when one of our engineers mentioned that Direct Memberships are generally not optimal at scale. We figured that during the lifecycle of our project we might need to add 5000 arbitrary devices to a collection. What would happen then?

My colleague pointed to this article: http://rzander.azurewebsites.net/collection-scenarios Which discussed some of the pitfalls of Direct Memberships, but didn’t go into the details of why, or discuss what the optimal solution would be for our scenario.

I went to our NWSCUG meeting last week, and there was a knowledgeable Microsoft fella there so I asked him during Lunch. He mentioned that there were no on-going performance problems with Direct Membership collections, however there might be some performance issues when creating/adding to the collection, especially within the Console (Load up the large collection in memory, then add a single device, whew!). He recommended, of course, running our own performance analysis, to find out what worked for us.

OK, so the hard way…

The Test environment

So off to my Standard home SCCM test environment: I’m using the ever efficient Microsoft 365 Powered Device Lab Kit. It’s a bit big, 50GB, but once downloaded, I’ll have a fully functional SCCM Lab environment with a Domain Controller, MDT server, and a SCCM Server, all running within a Virtual Environment, within Seconds!

My test box is an old Intel Motherboard circa 2011, with a i7-3930k processor, 32GB of ram, and running all Virtual Machines running off a Intel 750 Series NVME SSD Drive!

First step was to create 5000 Fake computers. That was fairly easy with a CSV file and the SCCM PowerShell cmdlet Import-CMComputerInformation.  Done!

Using the native ConfigMgr PowerShell cmdlets

OK, lets write a script to create a new Direct Membership rule in ConfigMgr, and write some Device Objects to the Collection.

Unfortunately the native Add-CMDeviceCollectionDirectMembershipRule cmdlet, doesn’t support adding devices using a pipe, and won’t let us add more than one Device at a time. Gee… I wonder if *that* will affect performance. Query the Collection, add a single device, and write back to the server, for each device added. Hum….

Well the performance numbers weren’t good:

Items to add Number of Seconds to add all items
5 4.9
50 53

As you can see the number of seconds increased proportionally to the number of items added. If I wanted to add 5000 items, were talking about 5000 seconds, or an hour and a half. Um… no.

In fact a bit of decompiling of the native function in CM suggests that it’s not really designed for scale, best for adding only one device at a time.

Yuck!

The WMI way

I decided to see if we could write a functional replacement to the Add-CMDeviceCollectionDirectMembershipRule cmdlet that made WMI calls instead.

I copied some code from Kadio on http://cm12sdk.net (sorry the site is down at the moment), and tried playing around with the function.

Turns out that the SMS_Collection WMI collection has  AddMembershipRule() <Singular> and a AddMembershipRules() <multiple> function. Hey, Adding more than once one device at a time sounds… better!

<Insert several hours of coding pain here>

And finally got something that I think works pretty well:

Performance numbers look much better:

Items to add Number of Seconds to add all items
5 1.1
50 1.62
500 8.06
5000 61.65

Takes about the same amount of time to add 5000 devices using my function as it takes to add 50 devices using the native CM function. Additionally some code testing suggests that about half of the time for each group is being performed creating each rule ( the process {} block ), and the remaining half in the call to AddMembershipRules(), my guess is that should be better for our production CM environment.

Note that this isn’t just a PowerShell Function, it’s operating like a PowerShell Cmdlet. The function will accept objects from the pipeline and process them as they arrive, as quickly as Get-CMDevice can feed them through the pipeline.

However more testing continues.

-k

 

 

 

Advertisements

New script – Import Machine Objects from Hyper-V into ConfigMgr

Quick Post, been doing a lot of ConfigMgr OSD Deployments lately, with a lot of Hyper-V test hosts.

For my test hosts, I’ve been creating Machine Objects in ConfigMgr by manually entering them in one at a time (yuck). So I was wondering what the process is for entering in Machine Objects via PowerShell.

Additionally, I was curious how to inject variables into the Machine Object that could be used later on in the deployment Process, in this case a Role.

Up next, how to extract this information from VMWare <meh>.

MDT 2013 UberBug01 – MDAC and the Fast Machine

Well MDT 2013 Update 1 is out. Yea!

Time to test and evaluate to see if there are any regressions in my test and environment.

Wait… Something went wrong…

dism

Fast System

As I mentioned in an earlier blog post, I recently purchased a super fast Intel 750 SSD to make my MDT Build machine run faster. Blazing Fast

WP_20150507_19_51_57_Pro

You might think: “Wow, that’s super cool, everything would be so much better with a faster machine”

Essentially, yes, except when it’s faster than the Operating System can handle.  :^(

The Bug

When updating a deployment share you may get the following error message:

Deployment Image Servicing and Management tool
Version: 10.0.10240.16384
 
Image Version: 10.0.10240.16384
 
Processing 1 of 1 - Adding package WinPE-MDAC-Package~31bf3856ad364e35~amd64~~10.0.10240.16384
 
Error: 1726
 
The remote procedure call failed.
An error occurred closing a servicing component in the image.
Wait a few minutes and try running the command again.
 

Dism.log shows nothing interesting:

 
2015-07-15 13:55:00, Error                 DISM   DISM.EXE: DISM Package Manager processed the command line but failed. HRESULT=800706BE
2015-07-15 13:55:00, Error                 DISM   DISM Manager: PID=2364 TID=2424 Failed to get the IDismImage instance from the image session - CDISMManager::CloseImageSession(hr:0x800706ba)
2015-07-15 13:55:00, Error                 DISM   DISM.EXE:  - CDismWrapper::CloseSession(hr:0x800706ba)

I asked someone knowledgeable at Microsoft (MNiehaus), and he mentioned that he saw it a couple of times, but couldn’t repro it consistently. However, I could easily reproduce the problem on demand with my hydration/buildout scripts.

Turns out that there is a narrow case where this bug manifests:

  • If you add any optional components to WinPE within MDT
  • If you have a fast hard disk (like my Intel 750 SSD)
  • If you have <UseBootWim> defined in you settings.xml, it may get worse.

The fast disk is most likely why the Windows Product Group teams never saw this bug in testing.

Well, I provided a repro environment to the Windows Product Groups involved in this component, even letting them log into my machine to reproduce the issue.

Good news is that they were able to determine what the root cause is ( timing related during unload of various components ), and even provided me with a private fix for testing! The private passed!

Now the real fun begins, there is a legitimate challenge here, because the error exists in the Windows 10 Servicing Stack, and that stack is embedded *into* WinPE.wim on the ADK and Boot.wim on the OS Install disk.

How do you update these files with the updated servicing stack, it’s not a trivial matter. They could send out a KB QFE Fix, and let customers manually update the files manually with dism.exe, they could also repackage/rerelease the ADK Itself, or worst case wait till the next release of the Windows OS ISO images.

I have been monitoring status, and there are several team members working on the release issues, and even someone from Customer Support acting as a customer advocate. My work is done.

Work Around

In the mean time, you can work around the issue:

  • Removing optional components from MDT. :^(
  • Of course, you move to a machine with a slower disk.
  • I had some luck getting optional components added when I set <UseBootWim> to false in the settings.xml file.
  • Additionally, Johan has mentioned that he can get it to pass if the OS running MDT is Windows 10 ( I was running Windows Server 2012 R2 ).

For me, it was easy, I don’t use the MDAC components in my environment, so I just removed them from the settings.xml file. Lame, I know.

-k

More Deployment bugs to follow!

Some practical notes on WSUS – Early 2015

Had to rebuild my WSUS Server recently and I decided to write down some notes for how I setup my machine.

The environment

I created a simple Virtual Machine running Windows Server 2012 R2 on one of my Windows 8.1 Host machines. 2GB Dynamic Memory, and a 500GB Local Hard Disk work great.

I don’t use the WSUS machine for day to day updates of my clients, instead the server is setup only for Imaging, it works great as a cache when re-creating test images over and over, so I don’t have to download each time.

The configuration

I basically configure my environment to just download and auto-approve everything *except* drivers. I don’t need drivers in my Imaging Environment, and I have see some comments that Driver Management in WSUS is problematic anyways.

approvals

Then I set the Synchronization Schedule to run every day.

When creating my Images via MDT Litetouch, I can easily point to my WSUS Server by entering the line:

WSUSServer=http://PickettW:8530

Exclusions

There are two updates I block on the Server Side:

  • Install package(s) for IE9 & IE 10 – Since Windows Update will eventually install IE 11 anyways, there is no need to install IE 9 or IE 10, and no reason to install all the updates for these versions
  • .NET Framework version 4.0 – Since we will be eventually be installing .NET framework 4.5. Version 4.5 already includes version 4.0 anyways.

Supersedence

Say you have two installation packages, KB555555 and KB666666. They both fix different things, but they patch the *same* file: ntoskrnl.exe. If we install both packages, which one wins? Well Update packages understand when given two packages, which one supersedes the other. That also means that if KB666666 supersedes KB555555, then you don’t even need to install KB555555 because it is going to be replaced anyways.

There is a lot of work in the WSUS internal database to keep track of all of the interdependencies, and supersedence. To keep things running smoothly, I recommend performing a “WSUS Server Cleanup Wizard” occasionally to ensure that Superseded updates are getting declined in the database and not being offered to clients.

The Problems

First thing to check is to open up the ZTIWindowsUpdate.log file to see what installed and what didn’t after your installation. Occasionally you may start to encounter problems during your installation, bad packages, and other mysterious installation errors.

For most errors, running the “WSUS Server Cleanup Wizard” is one of the quickest things you can do to clean up the machine and remove some obvious errors on the Server.

If you need more help, it would be good to look at the c:\windows\WindowsUpdate.log file to see if it gives any more hints as to what the problem is.

One of my least favorite updates is the “Definition Update for Windows Defender”, this update has all kinds of problems, loading in my environment. The problem is that Microsoft updates this package about *once a day*. So if you spend time trying to narrow down the bad update, Microsoft will have pushed out an updated version just to confuse you.

definition

Best advice, if you encounter an error, and it’s blocking your reference image install, just block the specific update instance in WSUS and try again.

Sucks I know.

New for the Lab: Intel 750 Series SSD

Got a new toy for the build lab: a new Intel 750 Series SSD, prices were fairly reasonable about 1$/GB, as compared to more recent SATA SSD drives at $.50/GB.

Performance wise it should be much faster than my standard SATA SSD drives.

WP_20150507_19_51_57_Pro

I was a little surprised to see that the retail package included a DVD, I still have a workstation machine with a DVD reader, but the drivers were already on Windows Update, so the disk was not necessary.

I plugged it into my lab build machine with an ASUS Motherboard, and a Z87 chipset. At first, the drive wasn’t detected, so I upgraded the uEFI firmware and that got it working. <whew!> I wasn’t interested in purchasing a new Motherboard, so that was a close call.

I re-ran my reference Hydration System building out 9 WIM images for x86,x64,Server versions of Windows 6.1 (Win7), 6.3 (Win8.1), and 10.

Without any other optimizations, just replacing the drive where the *.vhdx files were during build, most complex image (Windows 7 SP1 x64), went from:

Before: 3 hours 2 minutes
After: 2 hours 38 minutes

About a 13% decrease in time, not bad, still got some more work to do on the machine to make it faster, perhaps moving the OS drive to the new SSD, and other caching :^)

 

 

Sysprep Windows 10 Build 10074 Fix

There have been some reported SysPrep errors in Windows 10 Build 10074. Something to do with AppX controls (of course) in the panther logs.

There is a work around that has been floating here and there, hopefully it’s only a temporary fix for Windows 10:

Fix:

  • Stop the ’tiledatamodelsvc’ Service (ensure it has *actually* stopped)
  • Stop the ‘staterepository’ Service (ensure it has *actually* stopped)
  • Prevent both services from starting again by modifying the ImagePath
  • Run Sysprep normally
  • Then restore the ImagePath for both services

Code:

I have updated my private LTISysprep.wsf script for MDT 2013 Update 1 (Preview) here:

http://mdtex.codeplex.com/SourceControl/latest#Templates.2013.u1/Distribution/Scripts/LTISysprep.wsf

One of the cool things about Codeplex, is that you can compare it with the previous version to see what I changed. :^)

Note that this fix should only be temporary, and it is my intention to delete this post in the near future when fixed, hopefully before RTM. :^) Welcome to the fun of Beta Software.

Security Week – Locking down your deployment

Got a little bit over zealous this week on security issues, so I’ve decided to make amends by documenting some security issues you should all be aware of when designing your Windows OS Deployment Solution.

Logging

First of all the good news, when MDT runs, it logs a lot of information, including variables and values. I can see when the OSDComputerName variable was set, and what the value was. But what about Passwords? Well, it turns out that MDT ZTIUtility.vbs will read each line when we write to the log and if the string contains the word “Password” MDT Will skip over that line and continue as a protection to prevent passwords from appearing in the log.

But, sometimes we actually need to write a notice to the log that does not include any actual secrets. So we use a hack to get around the “Password” string search, we change “Password” to “P@ssword” so it will write that line to the log.

oLogging.CreateEntry "Success P@ssword Key file written", LogTypeInfo

Additionally, If we were to run any MDT VBScripts with the Debug=TRUE defined, it will write out passwords to the log file, Which is helpful in debugging scenarios, but be aware that this is dangerous in production.

Local Variables

When MDT LiteTouch and ZeroTouch are running, we will need to read and store variables for use throughout the process, we also need to store variables in such a way that can persist across reboots. If we just store the variables in memory they will disappear at reboot. Additionally, if we store variables in the registry, they may disappear if we boot into the new OS like WinPE.

If we are running within the Task Sequencer these variables are stored by SCCM, and are stored in an encrypted fashion. Which is great, however, any script running within the Task Sequencer can call the SCCM engine and Dump out the variables as plain text.

When MDT Litetouch is running outside of the SCCM/SMS Stand Alone Task Sequencer, it will store variables in the c:\minint\smsosd\osdlogs\varaibles.dat (xml) file as plaintext.

A lot of work has gone into protecting this information, without sacrificing the necessary script access. What we figured out is that Client Only security is hard.

Secrets

MDT ZeroTouch and LiteTouch pay attention to some very special variables that are handled with care:

  • ADMINPASSWORD

  • USERDOMAIN USERID USERPASSWORD

  • DOMAINADMINDOMAIN DOMAINADMIN DOMAINADMINPASSWORD

  • ADDSUSERNAME – ADDSPASSWORD

  • SAFEMODEADMINPASSWORD

  • BDEPIN

  • TPMOWNERPASSWORD

  • PRODUCTKEY

For these variables MDT LiteTouch when writing to the varaibles.dat file will store the data “Encoded” as a Base64 string. Note that I didn’t say “Encrypted” since the scripts need to be able to extract out the data as plaintext.

Passwords

Wait a minute, does that mean that my Domain Password, and Local Administrator Password are stored locally *UnEncrypted* during the installation process? Unfortunately Yes.

Why is that? Well it’s a hard problem that is not limited to MDT LiteTouch. The Local Administrator Password, and Domain Password (used for domain join) are inserted into the unattend.xml file and passed to Windows Setup. Windows Setup can only store these passwords as Plain Text or as a Base64 Encoded string.

Good news is that Windows and MDT do a good job of cleaning up passwords and other security vulnerabilities at the end of installation. Just watch to ensure that MDT completes correctly. When you see the MDT Litetouch Summary screen, your secrets have been deleted.

Security Mitigation

There are things we can do to limit these security vulnerabilities during a MDT LiteTouch deployment.

1. If you are worried about putting your administrator password into the system, don’t use a strong password, instead use a simple password string like “P@ssw0rd” in order to log in and perform the installation during the State Restore phase.

When MDT is done, we can then do several things to make the local administrator account more secure:

  • If your machine has other users that are local Administrators, then we should disable the local administrator account using the command: “Net user administrator /active:No” command.
  • If you require the local administrator account then force the user to change the local password. There are also several good tools out there for changing passwords on the server.

2. Additionally, if you are worried about storing the credentials used to join the domain I would recommend postponing the Domain Join until later.

Some other ideas:

  • Use a “restricted” account to join machines to the domain. Something that doesn’t have permissions elsewhere.
  • Use offline domain joins.
  • Put any new machine into a restricted OU until you can verify the machine.

Image Creation

In my lab, I’m less worried about my passwords getting out, because the system is mostly isolated. However I do have some steps to keep my systems secure and not publish secure passwords unnecessarily.

  • On the host machine with the deployment share I create a local account called “MDTUser” with a random password.
  • The “MDTUser” account is not a member of any localgroup, and I remove it from the Users Group. The only share that’s its given permission to is the \\server\DepoymentShare$ share.
  • Then I write the password into the bootstrap.ini file for UserPassword, update the Deployment share to create a new LitetouchPE_x??.iso image file.
  • I use the LitetouchPE_x??.iso image file to boot up my virtual machines and perform the Deploy and Capture Task sequence.

Note that the LitetouchPE_x??.iso image is not shared out, and I wouldn’t put it up on a PXE server. Only the local Virtual Machines will have access to it.

Logging

Additionally, it’s a good idea to set the SLSHARE variable in your CustomSettings.ini file to point to a logging share. When MDT LiteTouch is done with the task sequence, it will copy any relevant logs to the path for analysis later.

However, We may want to limit access to this share, to prevent others from reading the logs. This could be important when performing upgrades with USMT, we may not want even the names of the files migrated to be publicly available.

One trick I do is to set up the logging folder with the following permissions:

  • Allow “NT AUTHORITY\SYSTEM” Full Control, This folder, subfolders, and files.
  • Allow “BUILTIN\Administrators” Full Control, This folder, subfolders, and files.
  • Allow “CREATOR OWNER” Full Control, Subfolders, and files only.
  • Allow “NT AUTHORITY\Authenticated Users” Create folders / append data, This folder only.

Note that only Administrators are given full control.
Any user can create a folder, but those same users can’t *READ* folders created by other users, only those folders created by the owner.

You can assign the permissions using the command:

cacls c:\logs /S:"D:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICIIO;FA;;;CO)(A;;0x100004;;;AU) "    

It’s a handy trick.

-k