MDT UberBug11 – Security vs Usability

(Haven’t posted in a while, been busy with my day job(s), travel, scripting, sleep :^).

Some of you have recently noticed that MDT 2013 Update 1 has changed the way it sets the permissions on the network shares when it creates them for the first time.

When you create a new deployment share in MDT, it will ask you what the network share should be, and the wizard will automatically create the necessary bindings for the share and your local path, including setting up the permissions. Super!

Capture

The old permissions used to be “Everyone” has full access, it’s now set to “CREATOR OWNER”. I was somewhat confused by this change, and seemed a bit arbitrary to me. I suspect that someone filed a bug against MDT thinking that locking down the deployment share in the most restrictive way possible would somehow be better, because you know… Security! Think of the security breaches at Home Depot, and Target. PKI. Oh the humanity.

Well, MDT deployment shares don’t really store sensitive information, if you *DO* store any sensitive information on a MDT deployment share, then you are doing it wrong.

But anyways, someone made the decision to lock down the share, although “CREATOR OWNER”, this is kind of confusing to me, only one user? Why not use local “Administrators” group, local administrators already have full access to the files. “CREATOR OWNER” might only give access to one of several local “Administrators”.

Additionally, “Everyone” isn’t really that bad, access files over the network, you are still limited to the “File” level permissions on each file, which are better IMHO, I can create a Logging directory with Create/Write permissions, and set everything else to “Everyone” Read, with “Administrators” “Full R/W”

See: https://keithga.wordpress.com/2015/01/06/security-week-locking-down-your-deployment/

Anyways, this new permissions change for MDT 2013 Update 1 hasn’t caused much of a problem, as most users can easily work around the issue by adding extra approved users to the share afterwards.

Bootstrap.ini

Got a question today about a missing DeployRoot varaiable in BootStrap.ini.

MDT uses BootStrap.ini in WinPE to remember where to find the DeploymentShare to do the heavy lifting.

Normally, when creating a new DeploymentShare, MDT will automatically update the DeployRoot variable in BootStrap.ini, however several users were observing that this was no longer getting updated.

I used my trusty ILSpy to disassemble “C:\Program Files\Microsoft Deployment Toolkit\Bin\Microsoft.BDD.PSSnapIn.dll” and look at the “Provider class”, what I observed is this code segment:

IniManager iniManager = new IniManager(deploymentPointSettings["UNCPath"] + "\\Control\\Bootstrap.ini");
iniManager.Write("Default", "DeployRoot", deploymentPointSettings["UNCPath"]);
asdfsdf

You can see here that MDT is attempting to open the Bootstrap.ini file and write the Path to the DeployRoot Value.

Note that MDT is trying to load the Bootstrap.ini file using the same “UNCPath”? I suspected that MDT was failing to open the file due to the restrictive “Creator Owner” permissions, Sure enough, I tried opening the file over the network and it failed. Found!

Work Around

After creating a new deployment share in MDT, be sure to go back and fix some of the defaults:

  • Change the permissions, Something more permissive like local “Adminsitrators”
  • Change the \\server\deploymentshare$\control\bootstrap.ini to include
    deployroot=\\server\deploymentshare$
  • <More to follow I as do more testing>

MDT Bug: 451130 (known issue)

-k

Deployment Gotcha – Get Windows 10 App doesn’t work with MDT

Did you know that you can download *.iso images of Windows 10 using the Get Windows 10 app?

http://www.microsoft.com/en-us/software-download/windows10

It’s a tool that will allow you to download *.iso images for deployment.

The only problem is that it doesn’t actually download *.iso images, instead, it downloads *.esd images, which are highly compressed *.wim files, that are encrypted/encoded. The tool then decrypts/decodes the *.esd file into *.wim files, and constructs a *.iso image for use.

Additionally, the *.wim file is still a highly compressed *.esd file, so legacy tools that leverage the WIMGAPI library like imagex.exe don’t understand it.  Dism works fine.

C:\>"c:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\amd64\DISM\imagex.exe" /info g:\getwin10\Windows\sources\install.wim
ImageX Tool for Windows
 Copyright (C) Microsoft Corp. All rights reserved.
 Version: 10.0.10011.16384

Error opening file [g:\getwin10\Windows\sources\install.wim].

An attempt was made to load a program with an incorrect format.
C:\>dism /get-wiminfo /wimfile:g:\getwin10\Windows\sources\install.wim

Deployment Image Servicing and Management tool
 Version: 6.3.9600.17031

Details for image : g:\getwin10\Windows\sources\install.wim
 Index : 1
 Name : Windows 10 Pro
 Description : Windows 10 Pro
 Size : 9,343,966,034 bytes

The operation completed successfully.

That means that you can’t import the images into MDT, because MDT uses the WIMGAPI libarary, and powershell commands might not work either.

I tried to export the *.wim file to another *.wim file, but DISM.exe doesn’t like doing that either.

As a work around you might be able to export the *.wim file to a *.vhd container, and re-capture back to *.wim file with normal compression.

Johan has been documenting esd to wim techniques on his blog.

http://deploymentresearch.com/Research/Post/399/How-to-REALLY-create-a-Windows-10-ISO-no-3rd-party-tools-needed

Otherwise, I hope the DISM team fixes this for TH2 :^) <hint> <hint>

 

 

MDT UberBug10 – Win10 OS Capture using SCCM is not supported

Got an interesting question yesterday about the Symantec Management Agent (SMA). Someone asked about running sysprep.exe under the system context, and someone else mentioned that execution of the sysprep.exe is *not* supported in the system context for Windows 10. I suspect this has to do with all the new fancy Metro/Modern/Universal apps that have plenty of junk that needs to be cleaned up by sysprep.

What is the System Context? Well normally when you run a program in Windows, you are running it under your user context. However If you go to the Task Manager, “Details” tab, you can see that there are dozens of background processes running, some with the user name “SYSTEM” these are specially designed *.exe programs that run in the background and perform numerous tasks.

task manager

The problem

There is no problem for MDT LiteTouch build and capture task sequences, however if you were to use SCCM to build and capture, now you may have a supportability problem, because SCCM runs tasks sequences in the system context. Whoops!

I have filed a bug on connect asking for a Supportability statement from Microsoft.

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1721165

In the mean time, please continue to use MDT LiteTouch for building your Windows 10 images.  :^)

-k

MDT UberBug09 – BitLocker broken on uEFI machines for Win10

Got a IM today from a Consultant who was having trouble creating an end to end deployment of Windows 10 for some Surface Pro 3 devices.   Hey, I love Surface Pro 3 devices ( I have 3 in my office :^), so I thought I would help out.

The error message was when he tried to enable BitLocker, but that turned out to be a red-herring.

The Error

He had enabled BitLocker Pre-Provisioning and was trying to enable the protectors at the end, but kept getting errors like

BLError

This PC Doesn’t support entering a BitLocker recovery password during startup. Ask your administrator to configure Windows Recovery Environment so that you can use BitLocker.

What does WinRE have to do with bitlocker? OK, we ran REAgentC.exe /Info to see what the status of Recovery was, and turns out that it was *NOT* installed.

OK Let’s run the REAgentC.exe /enable command to turn it on:

reagentc_enable

REAGENTC.EXE: Windows RE cannot be enabled on a volume with BitLocker Drive Encryption enabled.

BitLocker wont’ start because it’s missing WinRE, and WinRE won’t install because of BitLocker. Super!!

Well after digging some more, we found out from the panther logs that the REAgentC.exe command was attempting to copy the WinRE.wim file from the OS partition to the WinRE partition created by MDT’s ZTIDiskPart.wsf but failed because the size was too small.  Too small?

The Root Cause

I Went back and looked at some WinRE sizes from current and past OS’es:

  • Windows 7 SP1 x86 – 145,287,084
  • Windows 7 SP1 x64 – 169,213,970
  • Windows 8.1 Update x86 – 193,132,205
  • Windows 8.1 Update x64 – 236,122,267
  • Windows 10 x86 – 238,226,804
  • Windows 10 x64 – 302,808,595

Wow, the Windows 10 x64 WinRE image is *far* bigger than the other versions.

Well, sometimes MDT can use the LitetouchPE WIM file as a WinRE file, how does the Win 10 ADK look:

  • LiteTouchPE_x64.wim  – 305,928,022
  • LiteTouchPE_x86.wim – 246,698,134

Well, since the Surface Pro 3 will *only* use x64, we better account for at least 310MB on our Recovery Partition.

What does ZTIDiskPart.wsf allocate? Easy it’s hard coded (in Mega Bytes):

Const WINRE_DRIVE_SIZE = 300

Our 302MB image is just *not* going to fit on a 300MB partition. Nope.

Bug

I opened this bug against the small partition size back in October of 2014:

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1012326/ztiwinre-fails-to-install-winre-images-on-correct-recovery-partitions-in-uefi-scenarios

Additionally, can you up the size of WINRE_DRIVE_SIZE
in ZTIDiskPArt.wsf to something like 400?

If you would like MDT to install WinRE.wim for Windows 10 scenarios, please log into connect and up-vote this connect bug.  :^)

 

-k

ZTIWinRE.wsf

P.S. Additionally, please do *NOT* ask me about ZTIWinRE.wsf, it’s a piece of junk and should be completely removed from MDT. I will not speak of it (unless you are on the MDT team, and I will be happy to explain why it is evil and wrong, and should be deleted ). :^p

MDT UberBug08 – Task Sequencer Crash

My 3rd bug related to the new Stand Alone Task Sequencer has to do with a Crash in the product. I’ve seen this pretty consistently when hydrating/building out my own SCCM 2012 Server.

At the end of the task sequence I will see the following:

error

SMSTS.Log shows:

<![LOG[Process completed with exit code 2147942402]LOG]!><time="23:36:24.402+420" date="07-12-2015" component="TSMBootstrap" context="" type="1" thread="3864" file="commandline.cpp:1123">
<![LOG[Exiting with return code 0x80070002]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="1" thread="3864" file="tsmbootstrap.cpp:1238">
<![LOG[shSmsTsKeyForScope.Open (HKEY_LOCAL_MACHINE, sSmsTsKeyForScope.c_str(), KEY_READ), HRESULT=80070002 (e:\nts_sccm_release\sms\framework\tscore\environmentscope.cpp,311)]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="0" thread="3864" file="environmentscope.cpp:311">
<![LOG[Failed to open key Software\Microsoft\SMS\47006C006F00620061006C005C007B00350031004100300031003600420036002D0046003000440045002D0034003700350032002D0042003900370043002D003500340045003600460033003800360041003900310032007D00\SMSTS]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="3" thread="3864" file="environmentscope.cpp:311">
<![LOG[GetKeyFromRegistry (m_sScopeRegKey.c_str(), Key), HRESULT=80070002 (e:\nts_sccm_release\sms\framework\tscore\environmentscope.cpp,764)]LOG]!><time="23:36:24.417+420" date="07-12-2015" component="TSMBootstrap" context="" type="0" thread="3864" file="environmentscope.cpp:764">

It’s a bit troubling, as I mentioned earlier with my other bugs related to the Stand Alone Task Sequencer, the Task Sequencer in MDT should be “rock solid”, and is a mission critical component, the fact that it crashes due to missing or corrupt registry is a problem.

I have filed a bug against MDT 2013.

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1712354

I have no work around, but thought I would share in case anyone else encounters the same error.

MDT 2013 UberBug07 – Not enough space on disk.

Here are some quick notes on a different kind of bug with MDT that I’m posting if anyone encounters it.

MDT 2013 Update 1 has a small bug in it that can cause refreshes from previous MDT deployments to fail.

It’s a complex interaction bug with MDT, the System and Recovery partitions, and WinRE.wim.

Symptoms

What happens is that when you try to “refresh” a machine from Windows 8.1 to another OS, MDT will attempt to place a copy of WinPE.wim on the “System” partition. However Windows 8.1 Setup may have already placed the WinRE.wim file on the System partition (rather than the WinRE partition where it should have been placed). This can happen even though you may not have enabled the ZTIWinRE.wsf script in your task sequence, because Windows 8.1 will always copy the WinRE.wim file to a non “OS/Boot” partition during setup.

When MDT tries to copy the WinPE.wim file to the local disk, it will first try to copy to the System partition, but will fail, because there is not enough space (thanks to the WinRE.wim file hogging up all the space). And you will get the error message:

Not enough space for boot image on boot partition using system partition

 

Fix

I filed a bug against MDT a while back:

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1136099

IN the mean time, You can view my private fix at:

http://mdtex.codeplex.com/SourceControl/diff/file/view/39534?fileId=Templates.2013.u1%2FDistribution%2FScripts%2FLTIApply.wsf

Lines 232 – 245

I basically delete the WinRE.wim file from the system partition. the older WinRE.wim file won’t be used in the new OS anyways, so it’s OK to delete. :^)

MDT UberBug05 – Batch scripting is dead.

Found this bug on the TechNet social group for MDT:

https://social.technet.microsoft.com/Forums/en-US/8b471021-4a84-4366-9467-acf6d03cba38/mdt-2013-update-1-batch-files-are-searched-in-cwindowssystem32-not-in-application-directory

Basically if you call a *.bat or *.cmd file in MDT as an Application, MDT will do some special processing to ensure that the default directory is in a good location for the bat or cmd file.

If not oNode.selectSingleNode("WorkingDirectory") is nothing then
  sDir = oUtility.SelectSingleNodeString(oNode,"WorkingDirectory")
  If Trim(sDir) <> "" and Trim(sDir) <> "." then
    If Left(sDir, 2) = ".\" then

      If (Instr(1, sCmd, ".CMD", 1) > 0 or Instr(1, sCmd, ".BAT", 1) > 0) and oEnvironment.Item("ResourceDrive") <> "" then
        If oEnvironment.Item("DeploymentMethod") = "MEDIA" then
          sDir = oEnvironment.Item("ResourceRoot") & Mid(sDir, 2)
        Else
          sDir = oEnvironment.Item("ResourceDrive") & Mid(sDir, 2)
        End If
     Else
       sDir = oEnvironment.Item("ResourceRoot") & Mid(sDir, 2)
    End if
  End if

  sDir = oEnvironment.Substitute(sDir)
  oUtility.ValidateConnection sDir
  oLogging.CreateEntry vbTab & vbTab & "Change directory: " & sDir, LogTypeInfo
  On Error Resume Next
  oShell.CurrentDirectory = sDir

ResourceDrive is a drive letter (Z:\) mapped to the Deployment share rather than the network path like ( \\MDT1\DeploymentShare$ ) Batch scripts like the drive letter better than the full path.

However in the RTM version of MDT that special test doesn’t work since sCMD is always empty for the block above.

Haven’t had a chance to code a fix, sadly it doesn’t directly affect me right now, I don’t use any cmd scripts.

If you want to upvote for a fix:

https://connect.microsoft.com/ConfigurationManagervnext/Feedback/Details/1682714

MDT Gotcha – No Progress is good progress.

Quick note:

from the MDT release notice:

http://blogs.technet.com/b/msdeployment/archive/2015/08/17/mdt-2013-update-1-now-available.aspx

One of the changed items is:

Switched to using DISM for imaging processes (instead of deprecated ImageX)

Some people might think this is a good thing, but I actually think this fix is bad.

ImageX.exe is the old tool used by Windows Vista to capture and apply *.wim images. The tool was superseded by dism.exe /apply-image and /capture-image. Really it didn’t matter since they *both* used the same wimgapi.dll API calls.

Why is Dism.exe better? Well for capture scenarios, it makes the process run much faster, turns out there was a bug in imagex.exe that would cause it to stop processing for 15 minutes at the end of the capture. Well it would be better to use dism.exe for capturing.

*However…*

The problem with dism.exe is that MDT doesn’t have the ability to capture the progress output from the program so it can’t display the pretty progress on the Task Sequencing Progress Bar. For Capture images, that’s not a big deal, but for image apply it would be better to keep imagex.exe, since MDT can capture the output from the program and display progress for Apply. imagex.exe apply didn’t have the imagex.exe 15 minute wait at the end so there was no need for it there.

 

During MDT beta I sent feedback to the MDT team that we should revert back to ImageX.exe for captures, but they declined to fix.

Please feel free to “upvote” my bug on the Microsoft Connect Site if you agree:

https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1685836

-k

MDT UberBug04 – Litetouch Success may mean Failure

I once knew a guy named Ted who didn’t like it when the “Check Engine” light came on in his car. He fixed the problem by covering up the Check Engine light with some black electrical tape. “There, now I won’t be bothered by the light anymore.” “But that’s not how to fix that.” I said… <sigh>

One of the really cool things about MDT LiteTouch is the fact that +90% of the product is included with source, VBScript and PowerShell. Even the Console and the PowerShell Providers written in C# can be decompiled and checked if there is a serious error. That means that if there are any problems, most of them can be fixed with some code changes on site.

ZTI (most of SCCM) and UDI are also closed boxes, written in C/C++, if you have a problem with them, then you need to file a bug with Microsoft Product Support <ouch>.

About the only thing that is a black box in MDT LiteTouch is the Stand Alone Task Sequencing Engine. It’s written in C/C++, but as long as I have worked on MDT, it’s been rock solid, only failing in a few obscure scenarios.

The bug

I had some weird errors in my task sequences for a couple of Windows 10 deployments, the “Deployment Summary” screen showed success, but the install failed somehow, what happened?

I tried creating a task sequence with an extra step to reproduce the problem:

tsstep1

The step is simple, it runs cmd.exe, and promptly fails. If we define “Continue on Error” as “False”, any failure returned in this step means that the *entire* Task Sequence should fail.

tsstep2

The result is that you get the Red (Or salmon) screen at the end of deployment.  That means that if I were to run this Task Sequence, I should see an error in MDT Litetouch.

However, what I see in my deployment is “Success”, not “Failure”

result1

What’s frustrating, is that the smsts.log is no help, it doesn’t show that the cmd.exe /c exit 42 command ran, it also doesn’t show what the error returned is in smsts.log.

...
Reading  logging settings from TS environment to set TS logging
Process completed with exit code 0
Exiting with return code 0x00000000
Used smsts.ini to set logging settings.
...

I can reproduce this problem quite easily in my Hydration/Buildout System, and I even provided a repro VHDX file to the MDT Product group team, but so far this bug has been marked as “Low Priority” and postponed till the next release.

Generally, I can work around most of the bugs in MDT, I just fix the script and move on, but if the error in a black box, with no work around there is not much I can do. <sigh>

That combined with MDT UberBug03 creates a dangerous scenario. Now we have task sequences that are failing, randomly, but returning success, and no logging to tell when and where it happened. This is a clear bug, and a regression from MDT 2013.

These two bugs alone (not to mention the other UberBugs I will be filing in the next few days), lead me to believe that MDT has some serious problems that could lead to problems in production.

Please let me know if you see this in your deployments. And upvote my Connect bug.

Bug: 1685295

MDT 2013 UberBug03 – Less Logging is not better

During the announcement of MDT 2013 Update 1 yesterday it was mentioned that one of the new features is:

Updated task sequence binaries from System Center 2012 R2 Configuration Manager SP1

Deployment pros know what a Task Sequence is, it’s the steps performed in order that make up a full deployment.

ts

Even when running LiteTouch, MDT will use System Center binaries to do the heavy lifting of calling each program, managing variables, and logging the results.

Logging

During task sequence execution, the SCCM Stand Alone Task Sequencing Engine will log each step, and log the results, most of the time in %temp%\smsts.log. For example, say we add a bad step to our task sequencer. In our SMSTS.log file would we see:

!------------------------------------------!
Expand a string: WinPEandFullOS
Executing command line: cmd.exe /c Echo Hello TS & exit 28
Process completed with exit code 28
!------------------------------------------!
Failed to run the action: Run Command Line.
The printer is out of paper. (Error: 0000001C; Source: Windows)

That’s great, I created a step in the Task Sequencer that retuned back a non-zero error code, and the step was not marked as “continue on error”, so the Task Sequence stopped because of it. At least we know what step failed.

Updated for MDT 2013 Update 1

Now with MDT 2013 Update 1, they have updated the core Task Sequencing binaries, and there is new functionality. The new functionality is to *not* log the steps, even when there is an error.

No Logging, this sucks.

This means that if one of your steps fails, and returns a non-zero result, the MDT Task Sequence may fail, and you the MDT Administrator may not be able to figure out why or where it failed.

I consider this to be a major flaw of the New Task Sequencing engine. I have filed a bug against the MDT team a while back, but they have indicated that they won’t look at a fix until the fall 2015.

If you think you can create task sequences that never have problems, then you should be OK, but logging after the fact is an important part of any *Production* MDT environment, as it blocks debugging.

There is no known work around.