Deployment Gotcha – Get Windows 10 App doesn’t work with MDT

Did you know that you can download *.iso images of Windows 10 using the Get Windows 10 app?

http://www.microsoft.com/en-us/software-download/windows10

It’s a tool that will allow you to download *.iso images for deployment.

The only problem is that it doesn’t actually download *.iso images, instead, it downloads *.esd images, which are highly compressed *.wim files, that are encrypted/encoded. The tool then decrypts/decodes the *.esd file into *.wim files, and constructs a *.iso image for use.

Additionally, the *.wim file is still a highly compressed *.esd file, so legacy tools that leverage the WIMGAPI library like imagex.exe don’t understand it.  Dism works fine.

C:\>"c:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\amd64\DISM\imagex.exe" /info g:\getwin10\Windows\sources\install.wim
ImageX Tool for Windows
 Copyright (C) Microsoft Corp. All rights reserved.
 Version: 10.0.10011.16384

Error opening file [g:\getwin10\Windows\sources\install.wim].

An attempt was made to load a program with an incorrect format.
C:\>dism /get-wiminfo /wimfile:g:\getwin10\Windows\sources\install.wim

Deployment Image Servicing and Management tool
 Version: 6.3.9600.17031

Details for image : g:\getwin10\Windows\sources\install.wim
 Index : 1
 Name : Windows 10 Pro
 Description : Windows 10 Pro
 Size : 9,343,966,034 bytes

The operation completed successfully.

That means that you can’t import the images into MDT, because MDT uses the WIMGAPI libarary, and powershell commands might not work either.

I tried to export the *.wim file to another *.wim file, but DISM.exe doesn’t like doing that either.

As a work around you might be able to export the *.wim file to a *.vhd container, and re-capture back to *.wim file with normal compression.

Johan has been documenting esd to wim techniques on his blog.

http://deploymentresearch.com/Research/Post/399/How-to-REALLY-create-a-Windows-10-ISO-no-3rd-party-tools-needed

Otherwise, I hope the DISM team fixes this for TH2 :^) <hint> <hint>

 

 

MDT UberBug10 – Win10 OS Capture using SCCM is not supported

Got an interesting question yesterday about the Symantec Management Agent (SMA). Someone asked about running sysprep.exe under the system context, and someone else mentioned that execution of the sysprep.exe is *not* supported in the system context for Windows 10. I suspect this has to do with all the new fancy Metro/Modern/Universal apps that have plenty of junk that needs to be cleaned up by sysprep.

What is the System Context? Well normally when you run a program in Windows, you are running it under your user context. However If you go to the Task Manager, “Details” tab, you can see that there are dozens of background processes running, some with the user name “SYSTEM” these are specially designed *.exe programs that run in the background and perform numerous tasks.

task manager

The problem

There is no problem for MDT LiteTouch build and capture task sequences, however if you were to use SCCM to build and capture, now you may have a supportability problem, because SCCM runs tasks sequences in the system context. Whoops!

I have filed a bug on connect asking for a Supportability statement from Microsoft.

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1721165

In the mean time, please continue to use MDT LiteTouch for building your Windows 10 images.  :^)

-k

MDT UberBug09 – BitLocker broken on uEFI machines for Win10

Got a IM today from a Consultant who was having trouble creating an end to end deployment of Windows 10 for some Surface Pro 3 devices.   Hey, I love Surface Pro 3 devices ( I have 3 in my office :^), so I thought I would help out.

The error message was when he tried to enable BitLocker, but that turned out to be a red-herring.

The Error

He had enabled BitLocker Pre-Provisioning and was trying to enable the protectors at the end, but kept getting errors like

BLError

This PC Doesn’t support entering a BitLocker recovery password during startup. Ask your administrator to configure Windows Recovery Environment so that you can use BitLocker.

What does WinRE have to do with bitlocker? OK, we ran REAgentC.exe /Info to see what the status of Recovery was, and turns out that it was *NOT* installed.

OK Let’s run the REAgentC.exe /enable command to turn it on:

reagentc_enable

REAGENTC.EXE: Windows RE cannot be enabled on a volume with BitLocker Drive Encryption enabled.

BitLocker wont’ start because it’s missing WinRE, and WinRE won’t install because of BitLocker. Super!!

Well after digging some more, we found out from the panther logs that the REAgentC.exe command was attempting to copy the WinRE.wim file from the OS partition to the WinRE partition created by MDT’s ZTIDiskPart.wsf but failed because the size was too small.  Too small?

The Root Cause

I Went back and looked at some WinRE sizes from current and past OS’es:

  • Windows 7 SP1 x86 – 145,287,084
  • Windows 7 SP1 x64 – 169,213,970
  • Windows 8.1 Update x86 – 193,132,205
  • Windows 8.1 Update x64 – 236,122,267
  • Windows 10 x86 – 238,226,804
  • Windows 10 x64 – 302,808,595

Wow, the Windows 10 x64 WinRE image is *far* bigger than the other versions.

Well, sometimes MDT can use the LitetouchPE WIM file as a WinRE file, how does the Win 10 ADK look:

  • LiteTouchPE_x64.wim  – 305,928,022
  • LiteTouchPE_x86.wim – 246,698,134

Well, since the Surface Pro 3 will *only* use x64, we better account for at least 310MB on our Recovery Partition.

What does ZTIDiskPart.wsf allocate? Easy it’s hard coded (in Mega Bytes):

Const WINRE_DRIVE_SIZE = 300

Our 302MB image is just *not* going to fit on a 300MB partition. Nope.

Bug

I opened this bug against the small partition size back in October of 2014:

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1012326/ztiwinre-fails-to-install-winre-images-on-correct-recovery-partitions-in-uefi-scenarios

Additionally, can you up the size of WINRE_DRIVE_SIZE
in ZTIDiskPArt.wsf to something like 400?

If you would like MDT to install WinRE.wim for Windows 10 scenarios, please log into connect and up-vote this connect bug.  :^)

 

-k

ZTIWinRE.wsf

P.S. Additionally, please do *NOT* ask me about ZTIWinRE.wsf, it’s a piece of junk and should be completely removed from MDT. I will not speak of it (unless you are on the MDT team, and I will be happy to explain why it is evil and wrong, and should be deleted ). :^p

New Windows 10 feature – Edition Upgrade – ChangePk.exe Gotchas

( TL;DR – Jump down to “The Problems” if you are ever thinking of running ChangePk.exe to change Windows 10 Versions)

One of the cool new features of Windows 10 is the ability to perform an “Edition Upgrade”. The ability to convert an existing Windows 10 instance from one version to another.

The collection of Windows 10 versions available for download is crazy:
(Not including the Phone, Server, IoT, and VHD versions)

  • Windows 10 with Bing
  • Windows 10 Home
  • Windows 10 Home for OEM’s
  • Windows 10 Pro
  • Windows 10 Pro for OEm’s
  • Windows 10 Enterprise
  • Windows 10 Enterprise – LTSB
  • Windows 10 Enterprise Eval
  • Windows 10 Enterprise Eval – LTSB
  • Windows 10 Education
  • Windows 10 Pro Checked

Multiply the SKU’s above to include both x86 and x64
Multiply the SKU’s above with the “N” version to comply with some import laws.
Multiply all the SKU’s above with different versions for each Language Type.
<whew> Keeping track of all these types  a chore (please let me know if I missed any :^)

Scenario

Say you purchase 150 new computers of different types (some Desktop, some Laptop, and some Tablets). You have a Enterprise Volume Licensing agreement with Microsoft, so you want all of these machines to run Enterprise VL. How do you get Enterprise on these machines.

The old Windows way was to create news images for the OS Verison you wanted, collect the drivers for the hardware types you had, and to create images or better yet create a MDT or SCCM deployment solution, It could take a day for development and another day or two to test for each hardware type.

Feature

Now with the new Edition Upgrade feature built in to Windows we now have a shortcut for this process. If we purchased our machines with Windows 10 Pro, then we just need to perform a single step to upgrade the version type from Pro to Enterprise.  This has a huge advantage since we can leverage the drivers already installed on the machine by each OEM.

There are a couple of deployment options available for you to perform the upgrade.

  • Manual – Start up Windows 10 “Settings” –> “Update & Security” –> “Activation” tab –> “Change product key”
    This will run “ChangePk.exe” to perform the actual updating the machine.
  • MDM – If you have an MDM, you can “push” out the Version change to the machine from the MDM host.
  • ICD – You can create a Windows Imaging and Configuration Designer ” (ICD) package with the appropriate new product Key, and apply it to Windows 10.
  • Script – You can also call the ChangePk.exe command directly and pass in the new product key on the command line.

All we need to do is to figure out how to run the command to perform the actual upgrade. I have been monitoring some email threads and web articles describing the feature, and decided to get down “into the weeds” and try myself.

I setup one of my Surface Pro 3 machines with the latest Windows 10 Pro recovery image. Ready to upgrade.

Upgrade Options

Going through the options available above, let’s pick an option to deploy.

  • Manual – We have 150 machines that we need to upgrade, and I would prefer not to perform these steps manually, instead I would prefer to do some quick and dirty scripting. :^)
  • MDM – I don’t have an MDM (Mobile Device Management) suite yet. (Sorry Intune)
  • ICD – ICD looks like a good option on paper, but after digging into the details, you can only apply the ICD package against an “offline” image, meaning you can apply it to a mounted WIM, or you will need to Boot into WinPE just to run this command, which is not optimal (I would prefer just to boot into Audit Mode, perform an edition update, and reseal):
DISM.exe /Image=C:\ /Add-ProvisioningPackage /PackagePath:C:\oem.ppkg
  • Script – Finally there is the option to run ChangePk.exe from the command line and pass in the new Product Key. For example to change pro to Enterprise:
Changepk.exe /productkey:NPPR9-FWDCX-D2C8J-H872K-2YT43

(in case you’re wondering, that is the Windows 10 Enterprsie GVLK key – publicly available/documented – that says “active via KMS”. it will change a Windows 10 Pro install to Windows 10 Enterprise and cause it seek out a KMS server after it is rebooted). Thanks to Mike Niehaus for pointing in the right direction.

The problems

First off, the ChangePk.exe command above didn’t work on my Surface Pro 3 device. I tried the usual discovery methods like calling “ChangePk.exe /?”, but returned nothing. I also tried searching the web, but Microsoft has no documentation on it ( as of 8/25/15, nearly a month after General Availability of Windows 10).

The ever Amazing Johan Arwidmark revealed that there is no “:” after /ProductKey. Cool, and he also revealed that there are also flags to suppress Reboots and to suppress UI.

Changepk.exe /ProductKey NPPR9-FWDCX-D2C8J-H872K-2YT43

However, that’s when things got really weird, when we were informed by someone in authority on the matter that the flags to suppress Reboots and UI not supported, not documented, subject to change, and we were asked not to use these flags… Oh.

Then I found out that once you initiate ChangePk.exe to perform a version upgrade, there is no stopping it, in fact when ready, it will *immediately* force a reboot of the local machine. How Immediate is the reboot? Well, lets hope you don’t have any work open on the local box, because you won’t be given a chance to save your work on any open files. Gone!

Automation and Scripting

We then had a … Lively … discussion on the support of OS components for automation and scripting.

First of all, let’s scope out the audience here. Windows 10 Pro to Enterprise Version Transformation should be a very popular option for some out there. And many IT departments will probably ask themselves about automation options when pressed with the 150 computer deployment scenario I imagined above.

It would be nice if the ChangePk.exe could suppress the UI, but OK, whatever, I guess it will have a UI.

But forced reboots are a problem, that means that it’s a horrible idea to run this command remotely, if a user was logged on, but away from their desk, they could come back to find all of their work gone. And help desk get’s an angry call.

I have been involved with companies that have had to terminate IT Administrators who sent out forced “reboots” to corporate wide company desktops for the wrong reasons. Whoops.

Further discussion reveals that fixing this behavior in the immediate term is not going to happen.

So if you are an IT administrator who needs to perform Windows Version Upgrade remotely, be aware; ensure that you politely allow the user to shutdown gracefully first before running the ChangePk.exe command.

Call to action

It would be better if Microsoft supported automation and scripting to help IT administrators (documentation would also be nice.) The server business has their act together with “Powershell” working as a central common scripting and administration core. Products like Nano Server are forcing Windows teams to ensure that each feature has remote and scripting paths available for IT Management.

As I describe in my earlier post on automating Application Installation: Applications/tools should follow a couple of basic rules to make automation/scripting in tools like SCCM and MDT easier:

  • Rule 1: Provide unattended installs
  • Rule 2: Do not display blocking UI
  • Rule 3: Do not exit until done
  • Rule 4: No rebooting

ChangePk.exe violates Rule #4 in a big way :^(

My name is Keith Garner, and I love to make separate Microsoft technologies “appear” as if were designed as a whole. ;^) Support your local Scripter.

MDT UberBug08 – Task Sequencer Crash

My 3rd bug related to the new Stand Alone Task Sequencer has to do with a Crash in the product. I’ve seen this pretty consistently when hydrating/building out my own SCCM 2012 Server.

At the end of the task sequence I will see the following:

error

SMSTS.Log shows:

<![LOG[Process completed with exit code 2147942402]LOG]!><time="23:36:24.402+420" date="07-12-2015" component="TSMBootstrap" context="" type="1" thread="3864" file="commandline.cpp:1123">
<![LOG[Exiting with return code 0x80070002]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="1" thread="3864" file="tsmbootstrap.cpp:1238">
<![LOG[shSmsTsKeyForScope.Open (HKEY_LOCAL_MACHINE, sSmsTsKeyForScope.c_str(), KEY_READ), HRESULT=80070002 (e:\nts_sccm_release\sms\framework\tscore\environmentscope.cpp,311)]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="0" thread="3864" file="environmentscope.cpp:311">
<![LOG[Failed to open key Software\Microsoft\SMS\47006C006F00620061006C005C007B00350031004100300031003600420036002D0046003000440045002D0034003700350032002D0042003900370043002D003500340045003600460033003800360041003900310032007D00\SMSTS]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="3" thread="3864" file="environmentscope.cpp:311">
<![LOG[GetKeyFromRegistry (m_sScopeRegKey.c_str(), Key), HRESULT=80070002 (e:\nts_sccm_release\sms\framework\tscore\environmentscope.cpp,764)]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="0" thread="3864" file="environmentscope.cpp:764">

It’s a bit troubling, as I mentioned earlier with my other bugs related to the Stand Alone Task Sequencer, the Task Sequencer in MDT should be “rock solid”, and is a mission critical component, the fact that it crashes due to missing or corrupt registry is a problem.

I have filed a bug against MDT 2013.

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1712354

I have no work around, but thought I would share in case anyone else encounters the same error.

MDT 2013 UberBug07 – Not enough space on disk.

Here are some quick notes on a different kind of bug with MDT that I’m posting if anyone encounters it.

MDT 2013 Update 1 has a small bug in it that can cause refreshes from previous MDT deployments to fail.

It’s a complex interaction bug with MDT, the System and Recovery partitions, and WinRE.wim.

Symptoms

What happens is that when you try to “refresh” a machine from Windows 8.1 to another OS, MDT will attempt to place a copy of WinPE.wim on the “System” partition. However Windows 8.1 Setup may have already placed the WinRE.wim file on the System partition (rather than the WinRE partition where it should have been placed). This can happen even though you may not have enabled the ZTIWinRE.wsf script in your task sequence, because Windows 8.1 will always copy the WinRE.wim file to a non “OS/Boot” partition during setup.

When MDT tries to copy the WinPE.wim file to the local disk, it will first try to copy to the System partition, but will fail, because there is not enough space (thanks to the WinRE.wim file hogging up all the space). And you will get the error message:

Not enough space for boot image on boot partition using system partition

 

Fix

I filed a bug against MDT a while back:

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1136099

IN the mean time, You can view my private fix at:

http://mdtex.codeplex.com/SourceControl/diff/file/view/39534?fileId=Templates.2013.u1%2FDistribution%2FScripts%2FLTIApply.wsf

Lines 232 – 245

I basically delete the WinRE.wim file from the system partition. the older WinRE.wim file won’t be used in the new OS anyways, so it’s OK to delete. :^)

MDT UberBug05 – Batch scripting is dead.

Found this bug on the TechNet social group for MDT:

https://social.technet.microsoft.com/Forums/en-US/8b471021-4a84-4366-9467-acf6d03cba38/mdt-2013-update-1-batch-files-are-searched-in-cwindowssystem32-not-in-application-directory

Basically if you call a *.bat or *.cmd file in MDT as an Application, MDT will do some special processing to ensure that the default directory is in a good location for the bat or cmd file.

If not oNode.selectSingleNode("WorkingDirectory") is nothing then
  sDir = oUtility.SelectSingleNodeString(oNode,"WorkingDirectory")
  If Trim(sDir) <> "" and Trim(sDir) <> "." then
    If Left(sDir, 2) = ".\" then

      If (Instr(1, sCmd, ".CMD", 1) > 0 or Instr(1, sCmd, ".BAT", 1) > 0) and oEnvironment.Item("ResourceDrive") <> "" then
        If oEnvironment.Item("DeploymentMethod") = "MEDIA" then
          sDir = oEnvironment.Item("ResourceRoot") & Mid(sDir, 2)
        Else
          sDir = oEnvironment.Item("ResourceDrive") & Mid(sDir, 2)
        End If
     Else
       sDir = oEnvironment.Item("ResourceRoot") & Mid(sDir, 2)
    End if
  End if

  sDir = oEnvironment.Substitute(sDir)
  oUtility.ValidateConnection sDir
  oLogging.CreateEntry vbTab & vbTab & "Change directory: " & sDir, LogTypeInfo
  On Error Resume Next
  oShell.CurrentDirectory = sDir

ResourceDrive is a drive letter (Z:\) mapped to the Deployment share rather than the network path like ( \\MDT1\DeploymentShare$ ) Batch scripts like the drive letter better than the full path.

However in the RTM version of MDT that special test doesn’t work since sCMD is always empty for the block above.

Haven’t had a chance to code a fix, sadly it doesn’t directly affect me right now, I don’t use any cmd scripts.

If you want to upvote for a fix:

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1682714

MDT Gotcha – No Progress is good progress.

Quick note:

from the MDT release notice:

http://blogs.technet.com/b/msdeployment/archive/2015/08/17/mdt-2013-update-1-now-available.aspx

One of the changed items is:

Switched to using DISM for imaging processes (instead of deprecated ImageX)

Some people might think this is a good thing, but I actually think this fix is bad.

ImageX.exe is the old tool used by Windows Vista to capture and apply *.wim images. The tool was superseded by dism.exe /apply-image and /capture-image. Really it didn’t matter since they *both* used the same wimgapi.dll API calls.

Why is Dism.exe better? Well for capture scenarios, it makes the process run much faster, turns out there was a bug in imagex.exe that would cause it to stop processing for 15 minutes at the end of the capture. Well it would be better to use dism.exe for capturing.

*However…*

The problem with dism.exe is that MDT doesn’t have the ability to capture the progress output from the program so it can’t display the pretty progress on the Task Sequencing Progress Bar. For Capture images, that’s not a big deal, but for image apply it would be better to keep imagex.exe, since MDT can capture the output from the program and display progress for Apply. imagex.exe apply didn’t have the imagex.exe 15 minute wait at the end so there was no need for it there.

 

During MDT beta I sent feedback to the MDT team that we should revert back to ImageX.exe for captures, but they declined to fix.

Please feel free to “upvote” my bug on the Microsoft Connect Site if you agree:

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1685836

-k

MDT UberBug04 – Litetouch Success may mean Failure

I once knew a guy named Ted who didn’t like it when the “Check Engine” light came on in his car. He fixed the problem by covering up the Check Engine light with some black electrical tape. “There, now I won’t be bothered by the light anymore.” “But that’s not how to fix that.” I said… <sigh>

One of the really cool things about MDT LiteTouch is the fact that +90% of the product is included with source, VBScript and PowerShell. Even the Console and the PowerShell Providers written in C# can be decompiled and checked if there is a serious error. That means that if there are any problems, most of them can be fixed with some code changes on site.

ZTI (most of SCCM) and UDI are also closed boxes, written in C/C++, if you have a problem with them, then you need to file a bug with Microsoft Product Support <ouch>.

About the only thing that is a black box in MDT LiteTouch is the Stand Alone Task Sequencing Engine. It’s written in C/C++, but as long as I have worked on MDT, it’s been rock solid, only failing in a few obscure scenarios.

The bug

I had some weird errors in my task sequences for a couple of Windows 10 deployments, the “Deployment Summary” screen showed success, but the install failed somehow, what happened?

I tried creating a task sequence with an extra step to reproduce the problem:

tsstep1

The step is simple, it runs cmd.exe, and promptly fails. If we define “Continue on Error” as “False”, any failure returned in this step means that the *entire* Task Sequence should fail.

tsstep2

The result is that you get the Red (Or salmon) screen at the end of deployment.  That means that if I were to run this Task Sequence, I should see an error in MDT Litetouch.

However, what I see in my deployment is “Success”, not “Failure”

result1

What’s frustrating, is that the smsts.log is no help, it doesn’t show that the cmd.exe /c exit 42 command ran, it also doesn’t show what the error returned is in smsts.log.

...
Reading  logging settings from TS environment to set TS logging
Process completed with exit code 0
Exiting with return code 0x00000000
Used smsts.ini to set logging settings.
...

I can reproduce this problem quite easily in my Hydration/Buildout System, and I even provided a repro VHDX file to the MDT Product group team, but so far this bug has been marked as “Low Priority” and postponed till the next release.

Generally, I can work around most of the bugs in MDT, I just fix the script and move on, but if the error in a black box, with no work around there is not much I can do. <sigh>

That combined with MDT UberBug03 creates a dangerous scenario. Now we have task sequences that are failing, randomly, but returning success, and no logging to tell when and where it happened. This is a clear bug, and a regression from MDT 2013.

These two bugs alone (not to mention the other UberBugs I will be filing in the next few days), lead me to believe that MDT has some serious problems that could lead to problems in production.

Please let me know if you see this in your deployments. And upvote my Connect bug.

Bug: 1685295

MDT 2013 UberBug03 – Less Logging is not better

During the announcement of MDT 2013 Update 1 yesterday it was mentioned that one of the new features is:

Updated task sequence binaries from System Center 2012 R2 Configuration Manager SP1

Deployment pros know what a Task Sequence is, it’s the steps performed in order that make up a full deployment.

ts

Even when running LiteTouch, MDT will use System Center binaries to do the heavy lifting of calling each program, managing variables, and logging the results.

Logging

During task sequence execution, the SCCM Stand Alone Task Sequencing Engine will log each step, and log the results, most of the time in %temp%\smsts.log. For example, say we add a bad step to our task sequencer. In our SMSTS.log file would we see:

!------------------------------------------!
Expand a string: WinPEandFullOS
Executing command line: cmd.exe /c Echo Hello TS & exit 28
Process completed with exit code 28
!------------------------------------------!
Failed to run the action: Run Command Line.
The printer is out of paper. (Error: 0000001C; Source: Windows)

That’s great, I created a step in the Task Sequencer that retuned back a non-zero error code, and the step was not marked as “continue on error”, so the Task Sequence stopped because of it. At least we know what step failed.

Updated for MDT 2013 Update 1

Now with MDT 2013 Update 1, they have updated the core Task Sequencing binaries, and there is new functionality. The new functionality is to *not* log the steps, even when there is an error.

No Logging, this sucks.

This means that if one of your steps fails, and returns a non-zero result, the MDT Task Sequence may fail, and you the MDT Administrator may not be able to figure out why or where it failed.

I consider this to be a major flaw of the New Task Sequencing engine. I have filed a bug against the MDT team a while back, but they have indicated that they won’t look at a fix until the fall 2015.

If you think you can create task sequences that never have problems, then you should be OK, but logging after the fact is an important part of any *Production* MDT environment, as it blocks debugging.

There is no known work around.