A replacement for SCCM Add-CMDeviceCollectionDirectMembershipRule PowerShell cmdlet

TL;DR – The native Add-CMDeviceCollectionDirectMembershipRule PowerShell cmdlet sucks for adding more than 100 devices, use this replacement script instead.

How fast is good enough? When is the default, too slow?

I guess most of us have been spoiled with modern machines: Quad Xeon Procesors, couple hundred GB of ram, NVME cache drives, and Petabytes of storage at our command.

And don’t get me started with modern database indexing, you want to know what the average annual rainfall on the Spanish Plains are? If I don’t get 2 million responses within a half a second, I’ll be surprised, My Fair Lady.

But sometimes as a developer we need to account for actual performance, we can’t just use the default process and expect it to work in all scenarios to scale.

Background

Been working on a ConfigMgr project in an environment with a machine count well over ~300,000 devices. And we were prototyping a project that involved creating Device Collections and adding computers to the Collections using Direct Membership Rules.

Our design phase was complete, when one of our engineers mentioned that Direct Memberships are generally not optimal at scale. We figured that during the lifecycle of our project we might need to add 5000 arbitrary devices to a collection. What would happen then?

My colleague pointed to this article: http://rzander.azurewebsites.net/collection-scenarios Which discussed some of the pitfalls of Direct Memberships, but didn’t go into the details of why, or discuss what the optimal solution would be for our scenario.

I went to our NWSCUG meeting last week, and there was a knowledgeable Microsoft fella there so I asked him during Lunch. He mentioned that there were no on-going performance problems with Direct Membership collections, however there might be some performance issues when creating/adding to the collection, especially within the Console (Load up the large collection in memory, then add a single device, whew!). He recommended, of course, running our own performance analysis, to find out what worked for us.

OK, so the hard way…

The Test environment

So off to my Standard home SCCM test environment: I’m using the ever efficient Microsoft 365 Powered Device Lab Kit. It’s a bit big, 50GB, but once downloaded, I’ll have a fully functional SCCM Lab environment with a Domain Controller, MDT server, and a SCCM Server, all running within a Virtual Environment, within Seconds!

My test box is an old Intel Motherboard circa 2011, with a i7-3930k processor, 32GB of ram, and running all Virtual Machines running off a Intel 750 Series NVME SSD Drive!

First step was to create 5000 Fake computers. That was fairly easy with a CSV file and the SCCM PowerShell cmdlet Import-CMComputerInformation.  Done!

Using the native ConfigMgr PowerShell cmdlets

OK, lets write a script to create a new Direct Membership rule in ConfigMgr, and write some Device Objects to the Collection.

Unfortunately the native Add-CMDeviceCollectionDirectMembershipRule cmdlet, doesn’t support adding devices using a pipe, and won’t let us add more than one Device at a time. Gee… I wonder if *that* will affect performance. Query the Collection, add a single device, and write back to the server, for each device added. Hum….

Well the performance numbers weren’t good:

Items to add Number of Seconds to add all items
5 4.9
50 53

As you can see the number of seconds increased proportionally to the number of items added. If I wanted to add 5000 items, were talking about 5000 seconds, or an hour and a half. Um… no.

In fact a bit of decompiling of the native function in CM suggests that it’s not really designed for scale, best for adding only one device at a time.

Yuck!

The WMI way

I decided to see if we could write a functional replacement to the Add-CMDeviceCollectionDirectMembershipRule cmdlet that made WMI calls instead.

I copied some code from Kadio on http://cm12sdk.net (sorry the site is down at the moment), and tried playing around with the function.

Turns out that the SMS_Collection WMI collection has  AddMembershipRule() <Singular> and a AddMembershipRules() <multiple> function. Hey, Adding more than once one device at a time sounds… better!

<Insert several hours of coding pain here>

And finally got something that I think works pretty well:

Performance numbers look much better:

Items to add Number of Seconds to add all items
5 1.1
50 1.62
500 8.06
5000 61.65

Takes about the same amount of time to add 5000 devices using my function as it takes to add 50 devices using the native CM function. Additionally some code testing suggests that about half of the time for each group is being performed creating each rule ( the process {} block ), and the remaining half in the call to AddMembershipRules(), my guess is that should be better for our production CM environment.

Note that this isn’t just a PowerShell Function, it’s operating like a PowerShell Cmdlet. The function will accept objects from the pipeline and process them as they arrive, as quickly as Get-CMDevice can feed them through the pipeline.

However more testing continues.

-k

 

 

 

Advertisements

New Tool – Disk Hogs

Edit: Heavily modified script for speed. Bulk of script is now running Compiled C# Code.

Been resolving some problems at work lately with respect to full disks. One of our charters is to manage the ConfigMgr cache sizes on each machine to ensure that the packages we need to get replicated, actually get replicated out to the right machines at the right time.

But we’ve been getting some feedback about one 3rd party SCCM caching tool failing in some scenarios. Was it really the 3rd party tool failing, or some other factor?

Well we looked at the problem and found:

  • Machines with a modest 120GB SSD Drive (most machines have a more robust 250GB SSD)
  • Configuration Manager Application Install packages that are around 10-5GB (yowza!)
  • Users who leave too much… crap laying around their desktop.
  • And several other factors that have contributed to disks getting full.

Golly, when I try to install an application package that requires 12GB to install, and there is only 10GB free, it fails.

Um… yea…

I wanted to get some data for machines that are full: What is using up the disk space? But it’s a little painful searching around a disk for directories that are larger than they should be.

Options

One of my favorite tools is “WinDirStat” which produces a great graphical representation of a disk, allowing you to visualize what directories are taking up the most space, and which files are the largest.  http://windirstat.net

Additionally I also like the “du.exe” tool from SysInternals.  https://live.sysinternals.com/du.exe

I wrap it up in a custom batch script file

@%~dps0du.exe -l 1 -q -accepteula %*

and it produces output that looks like:

PS C:\Users> dudir
    263,122 C:\Users\Administrator
      1,541 C:\Users\Default
  7,473,508 C:\Users\keith
      4,173 C:\Users\Public
  7,742,345 C:\Users
Files: 27330
Directories: 5703
Size: 7,928,161,747 bytes
Size on disk: 7,913,269,465 bytes

Cool, however, I wanted something that I could run remotely, and that would give me just the most interesting directories, say everything over 1GB, or something configurable like that.

So a tool was born.

Tool

The script will enumerate through all files on a local machine and return the totals. Along the way we can add in rules to “Group” interesting directories and output the results.

So, say we want to know if there are any folders under “c:\program files (x86)\Adobe\*” that are larger than 1GB. For the most part, we don’t care about Adobe Reader, since it’s under 1GB, but everything else would be interesting. Stuff like that.

We have a default set of rules built into the script, but you can pass a new set of rules into the script using a *.csv file ( I use excel )

Folder SizeMB
c:\* 500
C:\$Recycle.Bin 100
c:\Program Files 0
C:\Program Files\* 1000
C:\Program Files (x86) 0
C:\Program Files (x86)\Adobe\* 1000
C:\Program Files (x86)\* 1000
C:\ProgramData\* 1000
C:\ProgramData 0
C:\Windows 0
C:\Windows\* 1000
c:\users 0
C:\Users\* 100
C:\Users\*\* 500
C:\Users\*\AppData\Local\Microsoft\* 1000
C:\Users\*\AppData\Local\* 400

Example output:

The machine isn’t too interesting (it’s my home machine not work machine)

I’m still looking into tweaks and other things to modify in the rules to make the output more interesting.

  • Should I exclude \windows\System32 directories under X size?
  • etc…

If you have feedback, let me know

Script

Bypass OEM Setup and install your own image.

AutoPilot

Really Windows Autopilot is the future. As soon as the OEM’s get their act together, and offer machines without the bloatware and adware. Yea, I’m talking about you Anti-Virus Trial! Go away, shoo! Shoo! Give me Signature Images, or I’ll do it myself.

Unfortunately, I’m currently working for a client that is “Cloud Adverse”, and very… particular about Security. “have our machines go through the internet, and download our apps from a cloud, oh heavens no!!”.

So all machines come from the OEM’s and into a centralized distribution center, where they run a hodge-podge of OS Imaging tools to get the machines ready to ship out to each user.

And, No they don’t use any MDT… at least not yet…

Really it’s the Anti AutoPilot…

Where to start.

Well, when the machines arrive from the OEM, they are unboxed and placed on a configuration rack. If they are Desktop Machines, they are also connected to a KVM switch (Imagine several 8-port switches daisy chained together). Then they are plugged into power, network, and turned on.

Here’s our first challenge: How do we stop the PC from booting into the OEM’s OOBE process into OUR process? Well right now the technicians need to press the magic function key press at just the right time during boot up.

You know the drill, Press F12 for Dell, or perhaps press F9 for HP, or Press enter for Lenovo. Perhaps you have a Surface Device, and need to hold down the Volume button while starting the machine. Yuck, but better than nothing…

Well, the feedback we got from the technicians is that sometimes they miss pressing the button… at “just” the right time. This is really a problem for a Desktop PC’s connected to that KVM switch. If the Monitor doesn’t sync to the new PC quickly enough, you might easily miss pressing the boot override switch.

This sounded like a good challenge to start with.

Audit Mode

Really, IT departments don’t use Audit Mode. Audit Mode is a way to make customizations *during* Windows Setup and then re-seal the OS, so the end-user gets the nice shiny Windows Setup process (Specialize and OOBE) that they expect in a new PC.

Deployments in IT are all about bypassing the shiny Windows OOBE experience. No we don’t care about all the fancy new features in Cortana, We have already signed the SA agreement with Microsoft, we already know the domain to connect to, and our company has only one locale and keyboard type. IT departments would much rather skip all that, and get the user to their machine. So the thought of re-sealing a machine and going *back* to OOBE when we just finished joining to the domain and installing apps is silly.

But there are some Possibilities here. Turns out, that when Windows Setup is running, it will look for an Unattend.xml file and try to use it.

Methods for running Windows Setup

MDT uses an Unattend.xml file on the local machine it we can skip over the settings we know about, and re-launch MDT LiteTouch when finished. What about this process? If we place the Unattend.xml file on the root of a removable USB drive, the Windows version on the hard disk will look there and use these settings. The Lab Techs appeared to have a lot of USB sticks laying around, so using them shouldn’t be a problem.

We can’t use a MDT unattend.xml file as-is, but we can use AuditMode to get to a command prompt and install our own MDT LitetouchPE_x64.wim file.

  1. Boot into Audit Mode.
  2. While in Audit Mode, auto login using the Administrator Account.
  3. Find our PowerShell script and run it!

PowerShell script

Once we are in PowerShell, we now have full access to the system, and can modify it in any we choose. In this case, I have copied a LiteTouchPE_x64.wim file to the USB Stick, and we can force the Hard Drive to boot from that instead, continuing our process in MDT LiteTouch. Yea!

Now we have a bridge between the OEM system and our LiteTouch, or any other automated WinPE disk.

Yea! Now for the *REAL* automation to begin… 🙂

-k

 

Make DisMount-DiskImage work

TL;DR – DisMount-DiskImage doesn’t work the same it did in Windows Server 2012 R2, here is how to make it work in Windows 10 and Server 2016.

Dirty Boy

OK, with the release of Windows 1709, I’ve been downloading all sorts of *.ISO images from MSDN to try out the latest features, kits, and support utilities. As I develop scripts to auto mount and extract the contents, sometimes I leave the ISO images mounted, so I’ll need to clear everything off before I begin a new test run.

I developed a new powershell script function to do all of this for me: Dismount-Everything. I had the VHDX part working, but somehow the DVD part wasn’t working well.

I specifically wanted to dismount ISO images, even though I might now recall the path where they came from, even though the drive letter should be easily visible.

Blogs

I went to a couple of blog sites to find out how to dismount ISO images, and got some hits.

https://rcmtech.wordpress.com/2012/12/07/powershell-mounting-and-dismounting-iso-images-on-windows-server-2012-and-windows-8/

https://superuser.com/questions/499264/how-can-i-mount-an-iso-via-powershell-programmatically

But on the superuser site, the author suggests that you should be able to take the output from get-volume and pipe it into Get-DiskImage, but that was not working for me.

Get-Volume [Drive Letter] | Get-DiskImage | Dismount-DiskImage

No matter what, Get-DiskImage didn’t like the volume Path.

Get-Volume would output:

\\?\Volume{1f8dfd40-b7ae-11e7-bf06-9c2a70836dd4}\

but Get-DiskImage would expect:

\\.\CDROM1

Well, Windows NT will create virtual shortcuts between long path names like \\?\volume and something more readable like \\.\CDROM1, so I assumed there was an association there.

Well after testing I found out that this command didn’t work:

get-diskimage -devicepath \\?\Volume{1f8dfd40-b7ae-11e7-bf06-9c2a70836dd4}\

But this command did:

get-diskimage -devicepath \\?\Volume{1f8dfd40-b7ae-11e7-bf06-9c2a70836dd4}

Turns out that I just needed to strip out the trailing \.

Easy!

Code

Get-Volume | 
  Where-Object DriveType -eq 'CD-ROM' |
  ForEach-Object {
    Get-DiskImage -DevicePath  $_.Path.trimend('\') -EA SilentlyContinue
  } |
  Dismount-DiskImage

ZTISelectBootDisk.wsf new with BusType

Several years ago I wrote a script to help select which disk to deploy Windows to during your MDT LiteTouch or ZeroTouch task sequence.

https://keithga.wordpress.com/2013/09/18/ztiselectbootdisk-wsf/

Well, based on a request from my latest client, I have created a similar script that support BusType.

BackGround

My client is trying to install Windows Server 2016 on a Server with a SAN. When the machine boots to WinPE, one of the SAN drives appears *first* as Disk 0 (Zero). By default MDT Task Sequences will deploy to Disk Zero! My ZTISelectBootDisk.wsf already shows how to override. All we need to do is to find a way to tell MDT which disk to choose based on the correct WMI query.

Turns out it was harder than I thought.

What we wanted was the BusType that appears in the “Type” field when you type “Select Disk X” and then “detail disk” in Diskpart.exe.  When we ran “Detail Disk” in DIskpart.exe we could see the bus type: Fibre as compared to regular disks like SCSI or SAS.

The challenge was that the regular Win32_diskDrive WMI query wasn’t returning the BusType value, and we couldn’t figure out how to get that data through other queries.

I tried running some PowerShell queries like ‘Get-Disk’ and noticed that the output type was MSFT_Disk, from a weird WMI Namespace: root\microsoft\windows\storage. But adding that query to the script works! Yea!!!

BusType

What kind of BusTypes are there?

Name Value Meaning
Unknown 0 The bus type is unknown.
SCSI 1 SCSI
ATAPI 2 ATAPI
ATA 3 ATA
1394 4 IEEE 1394
SSA 5 SSA
Fibre Channel 6 Fibre Channel
USB 7 USB
RAID 8 RAID
iSCSI 9 iSCSI
SAS 10 Serial Attached SCSI (SAS)
SATA 11 Serial ATA (SATA)
SD 12 Secure Digital (SD)
MMC 13 Multimedia Card (MMC)
Virtual 14 This value is reserved for system use.
File Backed Virtual  15 File-Backed Virtual
Storage Spaces  16 Storage spaces
NVMe 17 NVMe

For this script we are *excluding* the following devices:

Name Value Meaning
Fibre Channel 6 Fibre Channel
iSCSI 9 iSCSI
Storage Spaces  16 Storage spaces
NVMe 17 NVMe

Meaning that the *FIRST* fixed device not in this list will become the new *Target* OS Disk. Run this query on your machine to see what disk will become the target:

gwmi -namespace root\microsoft\windows\storage -query 'select Number,Size,BusType,Model from MSFT_Disk where BusType <> 6 and BusTy
pe <> 9 and BusType <> 16 and BusType <> 17' | Select -first 1

Requirements

Reminder that this script requires MDT (latest), and the script should be placed in the %DeploymentShare%\Scripts folder. Additionally you should install all the Storage packages for WinPE, sorry I don’t recall *which* packages I selected when I did testing.

Script

-k

 

 

PowerShell Switch type never $null

Tales from the code review…

How do you test for a switch type in a PowerShell script?

How do you test for the *absence* of a switch in a PowerShell Script?

Came across this recently, and decided to dig into it further.

Script:

IN the example above, We have a function with a single switch argument. We then test against that argument, displaying “do something” if it’s set, and “Don’t do it” if not set.

Example Output:

PS C:\Users\Keith> C:\Users\Keith\Source\Example\test-switches.ps1
Don't do it
Don't do it
Do Something
Do Something

Cool!  Um… where did the “Never going to do it!” go? Well turns out that even when we don’t specify -test as an argument to the function, it’s still a switch defined as IsPresent = $false. So testing to see if it’s equal to $null will always fail, because it’s never $null.

 

Download Ignite 2017 videos locally

Thanks to Michel de Rooij on TechNet gallery for posting this slick script where you can download TechNet content locally to your machine.

https://gallery.technet.microsoft.com/Ignite-2016-Slidedeck-and-296df316

I wanted to select which videos to download, and wrote this powershell script to use out-gridview to download content. It calls the script above.

Usage:

You can run the command directly from powershell, just cut and paste this command:

iwr https://gist.githubusercontent.com/keithga/cb124fa3d2f96ac58470831c52d359a7/raw/8040ddaf971a27f0b35fd4b5e9c131048d29e8a5/get-Ignite2017Content.ps1 | % Content | Iex 

Comments:

  • Will download and cache the content locally so you can re-run the script repeatedly without having to wait to parse the website.
  • Will then display all the sessions in the PowerShell Out-GridView. Out-gridview is powerful.

  • Then will ask If you want to save the list to a *.html file for online viewing later.
  • Will also ask if you want to save the offline content to a local file.

The script: