Quick and Dirty Image Factory with MDT and PowerShell

I haven’t written a blog in a while, been busy with the new job at Tanium, but I did write this script recently, and thought I would share, in case anyone else found it interesting. Share it forwards.

Problem

Been working on solutions to upgrade Windows 7 to Windows 10 using Tanium as the delivery platform (it’s pretty awesome if I do say so my self). But as with all solutions, I need to test the system with some end to end tests.

As with most of my OS Deployment work, the Code was easy, the testing is HARD!

So I needed to create some Windows 7 Images with the latest Updates. MDT to the rescue! I created A MDT Deployment Share (thanks Ashish ;^), then created a Media Share to contain each Task Sequence. With some fancy CustomSettings.ini work and some PowerShell glue logic, I can now re-create the latest Windows 7 SP1 patched VHD and/or WIM file at moment’s notice.

Solution

First of all, you need a MDT Deployment Share, with a standard Build and Capture Task Sequence. A Build and Capture Task Sequence is just the standard Client.xml task sequence but we’ll override it to capture the image at the end.

In my case, I decided NOT to use MDT to capture the image into a WIM file at the end of the Task Sequence. Instead, I just have MDT perform the Sysprep and shut down. Then I can use PowerShell on the Host to perform the conversion from VHDX to WIM.

And when I say Host, I mean that all of my reference Images are built using Hyper-V, that way I don’t have any excess OEM driver junk, and I can spin up the process at any time.

In order to fully automate the process, for each MDT “Media” entry. I add the following line into the BootStrap.ini file:

    SkipBDDWelcome=YES

and the following lines into my CustomSettings.ini file:

    SKIPWIZARD=YES            ; Skip Starting Wizards
    SKIPFINALSUMMARY=YES      ; Skip Closing Wizards 
    ComputerName=*            ; Auto-Generate a random Computer Name
    DoCapture=SYSPREP         ; Run SysPrep, but don't capture the WIM.
    FINISHACTION=SHUTDOWN     ; Just Shutdown
    AdminPassword=P@ssw0rd    ; Any Password
    TASKSEQUENCEID=ICS001     ; The ID for your TaskSequence (Upper Case)

Now it’s just a matter of building the LitetouchMedia.iso image, mounting to a Hyper-V Virtual Machine, and capturing the results.

Orchestrator

What I present here is the Powershell script used to orchestrate the creation of a VHDX file from a MDT Litetouch Media Build.

• The script will prompt for the location of your MDT Deployment Share. Or you can pass in as a command line argument.
• The script will open up the Deployment Share and enumerate through all Media Entries, Prompting you to select which one to use.
• For each Media Entry selected, the script will
  • Force MDT to update the Media build (just to be sure)
  • Create a New Virtual Machine (and blow away the old one)
  • Create a New VHD file, and Mount into the Virtual Machine
  • Mount the LitetouchMedia.iso file into the Virtual Machine
  • Start the VM
• The script will wait for MDT to auto generate the build.
• Once Done, for each Media Entry Selected, the script will
  • Dismount the VHDx
  • Create a WIM file (Compression Type none)
  • Auto Generate a cleaned VHDx file

Code

The code shows how to use Powershell to:

• Connect to an existing MDT Deployment Share
• Extract out Media information, and rebuild Media
• How to create a Virtual Machine and assign resources
• How to monitor a Virtual Machine
• How to capture and apply WIM images to VHDx virtual Disks

Notes

I’ve been struggling with how to create a MDT VHDx file with the smallest possible size. I tried tools like Optimize-Drive and sDelete.exe to clear out as much space as possible, but I’ve been disappointed with the results. So here I’m using a technique to Capture the VHDx file as a Volume to a WIM file (uncompressed for speed), and the apply the Capture back to a new VHDx file. That should ensure that no deleted files are transferred. Overall results are good:

Before:   19.5 GB VHDx file --> 7.4 GB compressed zip
After:    13.5 GB VHDx file --> 5.6 GB compressed zip

Links

Gist: https://gist.github.com/keithga/21007d2aeb310a57f58392dfa0bdfcc2

https://wordpress.com/read/feeds/26139167/posts/2120718261

https://community.tanium.com/s/article/How-to-execute-a-Windows-10-upgrade-with-Tanium-Deploy-Setup

https://community.tanium.com/s/article/How-to-execute-a-Windows-10-upgrade-with-Tanium-Deploy-The-Sensors

https://community.tanium.com/s/article/How-to-execute-a-Windows-10-upgrade-with-Tanium-Deploy-Setup

 

Dell XPS 13 9360 Hardware Reset

TL;DR – If you are having some spontaneous errors start your laptop try disconnecting your batteries for an hour, and try again.

XPS 13 9360

Got a new laptop last month, it was time to replace the old one. Did some searching online and found something light, powerful, and at a good price. Dell XPS 13 9360:

•  8th Gen Intel® Core i7-8550U  (*Quad Core*)
• 512GB PCIe Solid Drive Drive (*NVMe Drive*)
• 16GB LPDDR3 1866MHz RAM
• 1x Thunderbolt port
• 13.3″ Touchscreen InfinityEdge QHD+ (3200 x 1800) Display

On sale at Costco for $1400. Overall a good value for a quad core laptop with NVMe.

The Break

Came back from a meeting (Starbucks? :)) Friday and the machine failed to boot. Got some display errors, rebooted, but got the recovery screen. So I shutdown for a while, when I rebooted, nothing. No Screen nothing.

However I did notice that the LED on the front was blinking, and I was able to catch the pattern, 2 and 7. Looking up in the service manual:

LCD Error!?!?! Crap.

A call to Dell Support confirmed the error, and a RMA ticket was generated, it could be two weeks before I get it back.

Battery

I wanted to archive the contents of the Disk before I sent it off to dell, so I got out my Torx screw driver.

But while I had the case open I disconnected the main Battery and the CMOS battery.

With most modern PC’s each of the components have small computers built in them. If they develop errors, do they reboot like the main OS when the power is off? If the battery is always connected that might not be true.  I had a similar problems recently with my SuperMicro test box, where flashing the BIOS wasn’t helping to resolve a complex problem I had with the box. Draining the CMOS battery and re-flashing the BIOS did work!

After an hour, I plugged in the batteries, and tried booting again. Yea, the machine works! It’s alive! I don’t have to send my machine in for repair.

Hopefully the machine will work a little bit longer than 45 days. We’ll know soon.

 

Silence is Golden during Setup

Thanks to @gwblok for pointing me to this twitter thread about Windows OOBE Setup.

Or rather, disable it first, and then enable it after OOBE is completed (in the task sequence). pic.twitter.com/ZCFTjr0lOs

— Johan Arwidmark (@jarwidmark) December 12, 2017

When Unattended is not Silent

During Windows 10 OOBE, the Windows Welcome process uses the Cortana voice engine to speak during Windows Setup.

Now we can go look for any updates

Shut up!

Yes, I’m one of those guys who sets my Sound Profile to “silent”, Silence is Golden!

And if I’m going to be running several Windows Deployments in my lab (read my home office), then I would prefer the machine to be silent. Reminds me of the XP/Vista days when we had boot up sounds. How rude.

So how to disable… Well the answer doesn’t appear to be that straight forwards.

SkipMachineOOBE

At first I suggested SkipMachineOOBE, and works on my test machine! Yea!

Then I got a reminder that SkipMachineOOBE is deprecated according to documentation.

DisableVoice

Thanks to @Jarwidmark for pointing me in the thread above to:

reg.exe add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\OOBE /v DisableVoice /t REG_DWORD /d 1

However, Microsoft Documentation also states that you should only use this for testing, and that Cortana Voice should be-enabled for users. OK… Fine, we’ll delete the key after setup is complete.

So where to place all this stuff?

Specialize

Several people suggested modifying the local registry within the imaging process, but I would prefer to avoid that, instead trying to see if we can perform the action during Setup using our unattend.xml file.

The command to disable would need to be *before* “OOBE”, sounds like the perfect job for the “Specialize” process.

Some quick testing, verified, and we are ready to go.

Automating OOBE

So, given the guidance from Microsoft on how to automate Windows 10:

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/settings-for-automating-oobe

Here are my changes:

• We disable Cortana during the Specialize Pass before OOBE.
• Then during OOBE, we clear the Cortana setting, and continue.

 

Windows 10 In-Place Security Considerations

TL;DR – When performing a Windows 10 In-Place upgrade, you must temporarily suspend any Disk Encryption protections, for BitLocker *AND* 3^rd party disk encryption too!

In Place Upgrade

So, how do we upgrade an Operating System? You know, the one we are currently using? Can we still upgrade it while still in use? Unfortunately, no. The Windows 10 In-Place process is very complex, and it requires full access to all the files on the machine. So how do we do that? Well, the upgrade process needs to shift to another OS, just temporarily, to modify the OS on our C:\ drive, we can use WinPE (Windows Pre-Installation Environment), or in this case WinRE (Windows Recovery Environment).

WinPE and WinRE are lightweight OS’es that are contained in a compressed boot.wim file, about 300MB to 500MB in size, and placed somewhere on the disk. We can boot into WinPE/RE and have it completely reside in memory. Now we have full access to the C:\ drive on the machine, and we can move files around and including a new OS.

3^rd Party Drivers

One of the challenges of shifting to a separate OS like WinPE/WinRE is that we’ll need to re-load any drivers required to access the system, including Disk and File System Drivers. For the most part, the latest versions of WinPE/WinRE will have very excellent support for the latest disk controller drivers. And it’s very rare that I’ve had needed to supply 3^rd party drivers for mainstream hardware. Starting with Windows 10 1607, Microsoft gives us the ability to add 3^rd party Drivers to the WinPE/WinRE using the /ReflectDrivers switch. This includes the ability to supply drivers for a Storage Controller or even a 3^rd party Disk Encryption tool. Anything that is required to access the machine.

Suspending Encryption

First some background…

At my house I have a Lock Box like this. I can place my house key in the box, and if someone needs to get into the house, I can just give them the code to the lock box. This is much better than giving everyone their own key, or just leaving the main door unlocked while I’m out. If I want to revoke access, I just change the code on the lock box, rather than re-keying my whole house.

If you have an OS disk that is encrypted, and you want to upgrade the OS, you probably don’t want to decrypt the ENTIRE disk before the OS upgrade, and re-encrypt the disk when the new OS is ready, that would take time to read and write data to the entire disk. Instead it would be better if we could leave the disk encrypted, and just temporarily give the setup system full access. It’s similar to the Lock box analogy above, we don’t want to give everyone access to the main encryption key, but the system will allow access at the right time to the right users.

For Microsoft BitLocker, the process is called “suspending”. We leave the disk encrypted, but the encryption keys for the disk are no longer protected. When the new OS is installed, we can re-establish protection via our usual protectors like TPM, SmartCard, Password, etc…

3^rd party encryption products need to function in the same way.  We would like to leave the disk encrypted, but any protections like “Pre-Boot authentication”, should be disabled, so the WinPE/WinRE Operating System, with the correct Encryption filter drivers have full access to the main OS. When finished, we can re-establish any Pre-Boot authentication protections supported by the encryption software like Passwords, TPM chips, Smart Cards, etc…  If the 3^rd party disk encryption product does not offer this then the WinPE/WinRE OS won’t be able to access the local C:\ Drive.

Misconceptions

I’ve been working with a client lately whose security team has correctly identified the In-Place Upgrade-Suspending Encryption behavior I described above. However, they incorrectly prescribe this as a vulnerability of BitLocker, and have not acknowledged that it is also a vulnerability of other 3^rd party disk encryption tools.

First off, yes, this is a known security Vulnerability in the way Windows 10 handles In-Place Upgrades, we simply must temporarily suspend protections as we move off to offline OS, this is by design. More below…

Secondly, It’s disingenuous to claim that this is only a BitLocker problem, by the design of the current Windows 10 In-Place upgrade system with the /ReflectDrivers hook, 3^rd party disk encryption tools must also suspend protections so the WinPE/WinRE offline OS’es.

This is really important for fully automated In-Place upgrade scenarios like MDT Litetouch or System Center Configuration Manger (SCCM) OSD (Operating System Deployment) tools.

Mitigations

Well, it’s not all gloom and doom, It’s not perfect, but like most things related to security, there are compromises, and tradeoffs.

Note that your data at-rest, protected by encryption, is only one potential threat vector where bad guys can get your data. There is also Malware, OS bugs, and other vectors that are made more secure with the latest Windows Releases. It *IS* important to keep your machine up to date and healthy with the latest OS and security tools, and simply avoiding upgrades because you don’t want to expose your machine, isn’t the best solution.

But there are also techniques/mitigations we can do to limit the exposure of your data during In-Place Upgrades. You will, of course, need to perform your own threat analysis. Some ideas might be:

• Don’t allow Upgrades to be performed in an automated fashion, always run attended. (not possible in some large environments).

• Only allow Upgrades to be performed on site, in semi-secured environments. Never over VPN or Wi-FI
• If you are running in a SCCM environment, we could develop some scripts/tools to monitor Upgrades. If a machine hasn’t returned from In-Place upgrade after XX minutes, then auto-open a Support Ticket, and immediately dispatch a tech.

-k

Install Windows 10 on Surface 1TB with MDT

TL;DR – Here is a script to get past the Disk0 to Disk2 mapping issue with the new Surface Pro with a 1TB drive.

Surface Hardware

OK, first a bit of history, I used to work for the Surface Imaging team back in 2015 to 2016. Overall a cool job, I learned a lot, and got to sharpen my PowerShell coding skills.

During that time I got to see my first Surface Studio device, even before it was released. Once of the unique features of the device was it’s unique disk architecture, it contains TWO disk drives, one a SSD in a M.2 format, and a Spinning Hard disk in a 2.5″ format. The OS contains a driver that uses the SSD as a cache. The idea is that you get the size of the 2TB hard disk, with (generally) the speed of the SSD.

Of course this creates a problem for OS deployment because we need to load the Special Caching driver into WinPE before deployment, so both drives are properly identified.

The Surface Pro with the 1TB drive is also unique in this manner, on the inside it isn’t a single 1TB drive, instead it’s two 512GB drives running in a Raid 0 configuration.

So you’re probably wondering how this works within WinPE on MDT, well the good news is that the built in 1709 drivers can correctly identify the two SSD disk as a single 1TB drive…

… The only problem is that it’s identified as Disk(2), and that breaks stuff.

ZTIDiskPart

Yes, yes, I know… mea culpa…

MDT (and SCCM/OSD) make an assumption on the “Format and Partition Disk” step: The target disk number is fixed for each Task Sequence. Now, we can change the target disk dynamically within the Task Sequence by chaning the OSDDiskIndex variable. But it will require some coding.

Fix 1

One fix, if you are OK with some WMI queries, is to test for a “Surface Pro” model and a 1TB disk at index 2. I would prefer to test for the ABSENCE of a disk at index 0, but not sure how to do that.

Fix 2

The following is a modification of my ZTISelectBootDisk.wsf script. Designed specifically for this configuration. Just drop it into the scripts folder and add a step in the Task Sequence before the “Format and Partition disk step.

Testing

Now this script has NOT been tested on a 1TB Surface device. However I *AM* available for testing the 1TB surface device. I can forward my home mailing address, if you want to send me one :^).

 

Create a Remote Desktop Conn. Mgr. file for Azure

Every now and then, I will spin up a test environment in azure with a couple of Virtual Machines. And as much as enjoy the User interface of the Portal, there have been times that I longed for the ease of use of the Remote Desktop Connection Manager.

One of the frustrations is that the *.rdp files downloaded from Azure run at full screen, which I find awkward, so the first thing I do is bring down the resolution, which is a pain, when I have several machines.

So I decided to create a tool to auto-generate a Remote Desktop Configuration Manager configuration file.

This also gave me the opportunity to play around with the way *.rdg stores passwords.

RDG security

First a word about security.

The Remote Desktop Connection Manager uses the “Data Protection API” to “Encrypt” passwords stored within the *.rdg file.

The great thing about this API, is that if another user were to open this file on another machine, it can’t be read. Only the user running on the same machine can extract this password.

Note that any program or script running under your user’s context can read this password as plaintext, works great for this script, but my one design change to the “Remote Desktop Connection Manager” would be to add a “PIN” or other layer of security abstraction to prevent other “rouge” processes or users from gaining access to the passwords stored locally on my machine.

Azure

This script will log into your Azure Account, enumerate all the service groups, and get a list of all Virtual Machines within these groups, Creating entries within the “Remote Desktop Connection Manager” for each Azure Virtual Machine.

Script

-k

Notes on Microsoft ADV170012 – TPM Madness.

Hidden within the latest Microsoft Security Advisory is a Whooper: ADV170012

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV170012

The summary is that some of the Infineon TPM chip implementations have a bug. And appears that someone has produced a Proof of Concept exploit. Wow.

Microsoft and Infineon have arguably done the right thing here and have announced the issue, produced a Hotfix to help customers better identify the issue, and have developed tools to update the issue through firmware.

What’s not clear to me is just what the issue is, and what the hotfix does. Unfortunately, it may be a while before Microsoft releases more information, while they give companies a head start working on application of the hotfixes.

A link in the article above suggest that exploit is not easy:

A successful attack depends on conditions beyond the attacker’s control. That is, a successful attack cannot be accomplished at will, but requires the attacker to invest in some measurable amount of effort in preparation or execution against the vulnerable component before a successful attack can be expected.

Which leads me to believe that any exploit is hard, requiring a highly skilled attacker, not someone who is going to steal my laptop from the local Starbucks in the hopes of getting my Credit Card number, saved somewhere on the machine.

Stay tuned…

Script

In the mean time, I decided to re-write the PowerShell script in the article above. The latest version works great when issuing commands remotely and collecting the data in a centralized location.

For example I could run the command:

icm { iwr 'https://gist.githubusercontent.com/keithga/22aa4500de40bc174f2f4921052e3b87/raw/Test-TPMReimann.ps1' | iex } -Cred $cred -Computer Pickett1,Pickett2,Pickett3,Pickett4 | ft *

And see the output:

Lucky for me I have Four machines that are affected with the Bad TPM Module.

• One machine is my work machine (Version 6.41)
• Two machines don’t have bad Infineon version numbers (Verison 3.19), but may need to be cleared anyways. Easy to do.
• One machine has the bad Infineon version (Version 4.32), but the TPM Module is on a replacement Riser card, and I can purchase a new one for $50.

Now to figure out how to address this at work.

Code

Notes

Don’t recall why I  named it Reimann, I think I saw that as a code word in an article somewhere, and it stuck. Is the name of the researcher who found the issue, or just an arbitrary code name?

Not sure why you need to know when the last Provision Time was on machines *without* the issue. Either the TPM chips works or not!?!?

 

 

Microsoft Groove RIP – Export your Playlist

OK… I’m using Groove. Don’t know why I paid the annual subscription, perhaps I had grand plans to sync up my music lists to a single platform and decided to give it a chance. Oh well… Microsoft just killed it.

Anyways, I’ve been collecting some songs over the past couple of years, and before I forget what they are, I thought I would export the list, only to find out that Groove only supports exporting to Spotify, well I don’t know what music service I’m planning on moving to, but it *might* not be Spotify, so I need to figure out how to export my list now.

I tried getting an Groove Music API, key, but Microsoft shutdown the service, I also tried another online service, but they wanted to charge a monthly fee. I did figure out that I can download my playlist locally to my machine. The files will be DRM protected, but I can use the file names to generate a playlist. How? Powershell to the rescue!

IF you would like to create a list, open up a powershell.exe command prompt and run the following command (Single line):

iwr https://gist.githubusercontent.com/keithga/8c3631beb2064cc33844505d97a76eb7/raw/e8f138929fdc54a9edf4b6ab58c0962f3c0d5a96/Export-GroovePlaylist.ps1 | % Content | IEX | export-csv -NoTypeInformation -path $env:USERPROFILE\desktop\myGrooveList.csv

This command will download the powershell script from GitHub, execute, and export to a file called MyGrooveList.csv on your desktop. ( or replace desktop with downloads, whatever).

Then you can open the MyGrooveList.csv file in Excel and import later.

Here is the full script:

Dell Latitude XX80 drivers won’t import into MDT Litetouch

TL;DR; Don’t import the Dell ControlVault driver into MDT Litetouch 8443 or older. It will crash the driver import.

Case of the mysterious…

David Landry posted a good question on the MDTOSD forum on http://myITForum.com”

Just curious what everyone / anyone has done with the Dell driver problem they have with the new Latitude XX80 series laptops. I imported their CAB files into MDT and got error.

It took me a while to narrow down the problem, but I was able to get a stack trace of the error:

System.Management.Automation.CmdletInvocationException: Length cannot be less than zero.
Parameter name: length ---> System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
 at System.String.Substring(Int32 startIndex, Int32 length)
 at Microsoft.BDD.PSSnapIn.InfInfo.CheckDriverFile(String filePath)
 at Microsoft.BDD.PSSnapIn.InfInfo..ctor(String infPath)
...

Looks like CheckDriverFile() function was trying to get a Substring() but the 2nd argument for Length was zero. Not much of a substring…

I opened my trusty Ilspy.exe and revealed:

Some further analysis reveals that the Dell ControlVault driver was the culprit, and I was able to dig down and find the file “Current_Version”. Current_Version has no File Extension, so it would cause the CheckDriverFIle() function to fail.

[cv_fw_copy]
bcmLynx_1.otp ;CV Firmware binaries
bcmLynx_7.otp
clearscd.bin
current_version
sbiLynxA0_1.otp
[...]

There is nothing technically wrong with the Dell Driver here, this is valid file. Additionally, this is the first time I have come across this bug in MDT, in the past 4 or so years this code has been active. It’s just an interoperability bug.

I have sent e-mail to both Microsoft and Dell. I’m hoping to qualify for the new Microsoft Bug Bounty program, but I’m not holding my breath. ;^)

 

Supermicro vs No_Drv

TL;DR – A NO_DRV driver is a driver package that doesn’t contain a driver binary. It uses the driver from another package (typically an in-box Microsoft driver).

Server Woes

One of my servers, a SuperMicro 5038A-IL has been locking up on me. I’ve been able to use the machine for the most part for the past year, but the upshot is that I can’t upgrade the machine from one Windows version to another. 22% of the way through WinRE (or sometimes WinPE), the machine has a hard Freeze, and won’t respond to mouse movement or keyboard. I’ve tried kernel debugging, hardware swaps, nothing works. My experience tells me this is a hardware problem. Time to escalate to SuperMicro support.

Upgrade your drivers

OK I should have seen this one coming, I call up support, and the first thing they ask me to do is make sure I have the latest Network and System Drivers.

1. WinRE doesn’t use Network, so this isn’t a problem.
2. System Drivers sure! Let’s download and check to see I have the latest stuff.

ftp://ftp.supermicro.com/driver/Intel_INF/Skylake_Series_Chipset/Chipset_v10.1.2.80.zip

No_Drv

I open the driver package, and take a look. Wait a minute, I don’t see any *.sys files, only *.inf and *.cat files.

sys – These are the compiled drivers, usually written in c/c++.
inf – These are the install files, typically there is at least one of these.
cat – This is the digial signature, signed by Microsoft to verify the source of the driver.

How can there be no driver in the driver package? Let’s look

[Version]
...
CatalogFile=Skylake.cat
...
[INTEL.NTamd64]
...
%PCI\VEN_8086&DEV_191FDesc%=Needs_NO_DRV,PCI\VEN_8086&DEV_191F

[Needs_NO_DRV]
Include=machine.inf
Needs=NO_DRV

Oh, NO_DRV, now I recall, this is an old trick. The device above DEV_191F doesn’t actually require a device driver, it’s transparently handled by Windows. But Windows doesn’t give it a descriptive name, just a generic name. So companies like Intel can provide this pack so you can give some devices more descriptive names in the Device Manager. The intel driver above points to an entry in the machine.inf driver already on the machine.

From:   “PCI Standard ISA bridge”

To:   “Intel(R) C226 Series Server Advanced SKU LPC Controller – 8C56”

That’s all it does, It doesn’t affect the operation of the OS or the hardware. Just the name! Sometimes this is helpful if you need to quickly identify a device, but, yes it’s mostly used for vanity, so I never load them on my machines.

So… I responded to SuperMicro, and confirmed I was using the latest drivers from intel. :^)