Security Week – Locking down your deployment

Got a little bit over zealous this week on security issues, so I’ve decided to make amends by documenting some security issues you should all be aware of when designing your Windows OS Deployment Solution.

Logging

First of all the good news, when MDT runs, it logs a lot of information, including variables and values. I can see when the OSDComputerName variable was set, and what the value was. But what about Passwords? Well, it turns out that MDT ZTIUtility.vbs will read each line when we write to the log and if the string contains the word “Password” MDT Will skip over that line and continue as a protection to prevent passwords from appearing in the log.

But, sometimes we actually need to write a notice to the log that does not include any actual secrets. So we use a hack to get around the “Password” string search, we change “Password” to “P@ssword” so it will write that line to the log.

oLogging.CreateEntry "Success P@ssword Key file written", LogTypeInfo

Additionally, If we were to run any MDT VBScripts with the Debug=TRUE defined, it will write out passwords to the log file, Which is helpful in debugging scenarios, but be aware that this is dangerous in production.

Local Variables

When MDT LiteTouch and ZeroTouch are running, we will need to read and store variables for use throughout the process, we also need to store variables in such a way that can persist across reboots. If we just store the variables in memory they will disappear at reboot. Additionally, if we store variables in the registry, they may disappear if we boot into the new OS like WinPE.

If we are running within the Task Sequencer these variables are stored by SCCM, and are stored in an encrypted fashion. Which is great, however, any script running within the Task Sequencer can call the SCCM engine and Dump out the variables as plain text.

When MDT Litetouch is running outside of the SCCM/SMS Stand Alone Task Sequencer, it will store variables in the c:\minint\smsosd\osdlogs\varaibles.dat (xml) file as plaintext.

A lot of work has gone into protecting this information, without sacrificing the necessary script access. What we figured out is that Client Only security is hard.

Secrets

MDT ZeroTouch and LiteTouch pay attention to some very special variables that are handled with care:

  • ADMINPASSWORD

  • USERDOMAIN USERID USERPASSWORD

  • DOMAINADMINDOMAIN DOMAINADMIN DOMAINADMINPASSWORD

  • ADDSUSERNAME – ADDSPASSWORD

  • SAFEMODEADMINPASSWORD

  • BDEPIN

  • TPMOWNERPASSWORD

  • PRODUCTKEY

For these variables MDT LiteTouch when writing to the varaibles.dat file will store the data “Encoded” as a Base64 string. Note that I didn’t say “Encrypted” since the scripts need to be able to extract out the data as plaintext.

Passwords

Wait a minute, does that mean that my Domain Password, and Local Administrator Password are stored locally *UnEncrypted* during the installation process? Unfortunately Yes.

Why is that? Well it’s a hard problem that is not limited to MDT LiteTouch. The Local Administrator Password, and Domain Password (used for domain join) are inserted into the unattend.xml file and passed to Windows Setup. Windows Setup can only store these passwords as Plain Text or as a Base64 Encoded string.

Good news is that Windows and MDT do a good job of cleaning up passwords and other security vulnerabilities at the end of installation. Just watch to ensure that MDT completes correctly. When you see the MDT Litetouch Summary screen, your secrets have been deleted.

Security Mitigation

There are things we can do to limit these security vulnerabilities during a MDT LiteTouch deployment.

1. If you are worried about putting your administrator password into the system, don’t use a strong password, instead use a simple password string like “P@ssw0rd” in order to log in and perform the installation during the State Restore phase.

When MDT is done, we can then do several things to make the local administrator account more secure:

  • If your machine has other users that are local Administrators, then we should disable the local administrator account using the command: “Net user administrator /active:No” command.
  • If you require the local administrator account then force the user to change the local password. There are also several good tools out there for changing passwords on the server.

2. Additionally, if you are worried about storing the credentials used to join the domain I would recommend postponing the Domain Join until later.

Some other ideas:

  • Use a “restricted” account to join machines to the domain. Something that doesn’t have permissions elsewhere.
  • Use offline domain joins.
  • Put any new machine into a restricted OU until you can verify the machine.

Image Creation

In my lab, I’m less worried about my passwords getting out, because the system is mostly isolated. However I do have some steps to keep my systems secure and not publish secure passwords unnecessarily.

  • On the host machine with the deployment share I create a local account called “MDTUser” with a random password.
  • The “MDTUser” account is not a member of any localgroup, and I remove it from the Users Group. The only share that’s its given permission to is the \\server\DepoymentShare$ share.
  • Then I write the password into the bootstrap.ini file for UserPassword, update the Deployment share to create a new LitetouchPE_x??.iso image file.
  • I use the LitetouchPE_x??.iso image file to boot up my virtual machines and perform the Deploy and Capture Task sequence.

Note that the LitetouchPE_x??.iso image is not shared out, and I wouldn’t put it up on a PXE server. Only the local Virtual Machines will have access to it.

Logging

Additionally, it’s a good idea to set the SLSHARE variable in your CustomSettings.ini file to point to a logging share. When MDT LiteTouch is done with the task sequence, it will copy any relevant logs to the path for analysis later.

However, We may want to limit access to this share, to prevent others from reading the logs. This could be important when performing upgrades with USMT, we may not want even the names of the files migrated to be publicly available.

One trick I do is to set up the logging folder with the following permissions:

  • Allow “NT AUTHORITY\SYSTEM” Full Control, This folder, subfolders, and files.
  • Allow “BUILTIN\Administrators” Full Control, This folder, subfolders, and files.
  • Allow “CREATOR OWNER” Full Control, Subfolders, and files only.
  • Allow “NT AUTHORITY\Authenticated Users” Create folders / append data, This folder only.

Note that only Administrators are given full control.
Any user can create a folder, but those same users can’t *READ* folders created by other users, only those folders created by the owner.

You can assign the permissions using the command:

cacls c:\logs /S:"D:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICIIO;FA;;;CO)(A;;0x100004;;;AU) "    

It’s a handy trick.

-k

Updated – PS2Wiz – Addition of Out-GridView support

I updated the PS2Wiz tool on CodePlex: http://PS2Wiz.CodePlex.com.

out-GridView

I’ve been using the PowerShell cmdlet Out-GridView for selecting and managing datasets in a graphical manner, and wanted to see if I could extend that functionality to the PS2Wiz tool. Additionally, I was a bit disappointed in the lack of ListBox support in the existing PS2Wiz implementation. The only features to allow a user to select an item from a list was the PromptForChoice() function as a collection of Radio Buttons.

What I ended up with is an override replacement of the Out-GridView cmdlet within PS2Wiz, meaning that when you call Out-GridView in your script, rather than calling the native PowerShell Out-GridView cmdlet with it’s own window, PS2Wiz redirects the command to the PS2Wiz version which displays the content in the Local Wizard Window.

For example if you were to create a PowerShell script for PS2Wiz that ran the command:

get-childitem c:\ | out-gridview

It would show in the wizard like:
CHI

You can also output an array of strings to Out-GridView and have it display as a list

type c:\windows\win.ini |out-gridview

list

The internal PS2Wiz version of Out-GridView supports all of the standard Out-GridView Parameters like -Wait -PassThru and -OutputMode. It’s a pretty good replacement for the internal Out-GridView cmdlet.

One exception is that the PS2Wiz version of Out-GridView supports passing in arrays for the -InputObject parameter. So the following command will list items in PS2Wiz, but not in regular PowerShell.exe

out-gridview -InputObject (get-Process |select-Object ID,Name)

MDTObject

Additionally, I added a treeview control for MDT Objects. This allows for some future MDT management scripts.


Import-Module "C:\Program Files\Microsoft Deployment Toolkit\bin\MicrosoftDeploymentToolkit.psd1"
New-PSDrive -Name "DS001" -PSProvider MDTProvider -Root "C:\DeploymentShare" | out-null
Select-MDTObject -OutputMode Multiple -RootNode "ds001:\Applications"

mdt

This allows me to develop some tools that manage MDT Shares, selecting Applications, Operating Systems, Drivers, Packages, TaskSequences, and more..

Could be useful. 🙂

New Tool – PS2Wiz – Run PowerShell scripts in a Wizard Host! http://ps2wiz.codeplex.com/

Posted my first project to CodePlex: PowerShell Host Wizard at http://ps2wiz.codeplex.com

It’s a way to distribute your PowerShell Scripts as an Executable (*.exe) with a Wizard User Interface.

Example1

It’s intended for those PowerShell developers who may be getting pressure to create fancy User Interfaces, but don’t want to get into the quagmire of developing custom Forms just as a front end to the PowerShell script that does the real work. PS2Wiz will automatically hook into the PowerShell system and construct the User Interface elements automatically when requested by the script.

* Write-Host – will become a label.
* Read-Host – will become a TextBox.
* Get-Credential – will display two TextBoxes for credentials.
* Copy with -Confirm – will display a radio button to confirm.
* Call a function with Mandatory Parameters – Those get auto generated.
* and more…

Video

I created a video to demonstrate how it works.

Link

http://ps2wiz.codeplex.com

Special Thanks to ikarstein for the excellent work on PS2Exe. Got me inspired to release it to CodePlex.
(And to @mikael_nystrom for pointing me to PS2Exe a few weeks ago via twitter)

Customizing Your Windows Deployments

Hello again from Mall of America in Minneapolis Minnesota at the Minnesota Management Summit 2014 (MMS for short). #mmsminnesota

Today I am giving a presentation on “Customizing Your Windows Deployments – Tips, Tricks, and Code”.

Links

Stick with Well-Known and Proven Solutions
MDT Wizard Studio

Code

I wanted share the source code examples from my presentation:

MMS2014 – Customizing Your Windows Deployments.zip

Inside you will find the following:

MMS2014 - Customizing Your Windows Deployments.pptx
1 - Unattend\makevhdx.ps1
1 - Unattend\make_client_vhd.cmd
1 - Unattend\Run-VHD.ps1
1 - Unattend\unattend.Workgroup.xml
1 - Unattend\unattend.xml
2 - ztigather\CustomSettings.ini
2 - ztigather\UserExit.vbs
3 - CMD\AppWrapper.cmd
3 - CMD\Source\7z925x64.msi
4 - VBS\ZTI_XXXXXXX.wsf
4 - VBS\ZTIUtility.vbs
5 - PS1\CM1.CleanUp.ps1
5 - PS1\Get-Service.ps1
5 - PS1\Test-MDTPowerShell.cmd
5 - PS1\Test-MDTPowershell.ps1

In the “1 – Unattend” folder is my example of how to take a Windows *.iso image and mount in Hyper-V.
In the “2 – ZTIGather” folder is a quick example of a customsettings.ini file and a userexit.vbs script.
In the “3 – CMD” folder is an example of using a CMD file to wrap commands in a batch script.
In the “4 – VBS” folder I have some examples on how to create MDT VBScript files.
In the “5 – PS1” folder I have an example of how to call a powershell script from within MDT.

-k

Imaging Factory performance

I’ve been experimenting recently with building a Hydration Imaging Factory on one of my servers. A Hydration Factory is a Windows Host that constructs Windows images for use in deployment.

Perhaps you have a simple setup in your environment using MDT LiteTouch. This could be something like a task sequence that installs Windows 7 x64, runs Windows Update, syspreps and captures back to a *.wim file. Or perhaps you have a laundry list of applications that need to be installed in your corporate standardized image for VDI scenarios. With the correct settings in your CustomSettings.ini file, this process could be fully automated, and repeatable. Spin up a Virutal Machine and 30 minutes later you have a new install.wim file.

A Hydration Imaging Factory will combine the automation of MDT LiteTouch with some PowerShell automation to build out a list of virtual machines.

Configuration

I’ve been spending some time trying to make my Hydration Factory system modular, and right now I can kick off a new build and my Host.
In my system:

  • All images are fully patched and have IE 11 and the KMDF
  • Some images are “Min” – No applications, just Updated/Patched
  • Some images are “Full” – Applications like Adobe Reader, Chrome, VCRT, etc.
  • I also create a Hyper-V specific versions (PersistAllDeviceInstalls)
  • I have packages for Office and SQL, but did not include below
  • I run a dism /clean command just before sysprep to trim the images
  • Results

    Given my host test machine (Simple single processor multi-core desktop, i7, 32GB of ram, and multiple SSD Drives). It took about 7 hours to build out the following Virtual Machines.


    4,080,851,813 WIN10STPX64.SRV.Full.HV.WIM
    4,080,982,651 WIN10STPX64.SRV.Full.WIM
    3,954,997,709 WIN10STPX64.SRV.Min.Core.HV.WIM
    3,955,009,907 WIN10STPX64.SRV.Min.Core.WIM
    3,955,336,881 WIN10STPX64.SRV.Min.HV.WIM
    3,955,175,443 WIN10STPX64.SRV.Min.WIM
    3,882,925,692 WIN10TPX64.ENT.Full.HV.WIM
    3,882,213,922 WIN10TPX64.ENT.Full.WIM
    3,754,245,946 WIN10TPX64.ENT.Min.HV.WIM
    3,754,582,199 WIN10TPX64.ENT.Min.WIM
    2,989,545,883 WIN10TPX86.ENT.Full.HV.WIM
    2,992,590,857 WIN10TPX86.ENT.Full.WIM
    2,921,467,219 WIN10TPX86.ENT.Min.HV.WIM
    2,921,762,549 WIN10TPX86.ENT.Min.WIM
    5,775,824,112 WIN2008R2SP1.Full.HV.WIM
    5,775,798,368 WIN2008R2SP1.Full.WIM
    4,618,522,652 WIN2008R2SP1.Min.HV.WIM
    4,618,521,668 WIN2008R2SP1.Min.WIM
    4,921,167,148 WIN2012R2U.Full.HV.WIM
    4,921,555,872 WIN2012R2U.Full.WIM
    4,513,623,325 WIN2012R2U.Min.Core.HV.WIM
    4,554,749,492 WIN2012R2U.Min.Core.WIM
    4,451,558,474 WIN2012R2U.Min.HV.WIM
    4,459,989,734 WIN2012R2U.Min.WIM
    5,955,716,470 WIN7SP1X64EVAL.Full.HV.WIM
    5,751,198,710 WIN7SP1X64EVAL.Full.WIM
    4,761,940,473 WIN7SP1X64EVAL.Min.HV.WIM
    4,776,248,329 WIN7SP1X64EVAL.Min.WIM
    4,223,192,736 WIN7SP1X86EVAL.Full.HV.WIM
    4,179,078,039 WIN7SP1X86EVAL.Full.WIM
    3,440,522,203 WIN7SP1X86EVAL.Min.HV.WIM
    3,440,523,165 WIN7SP1X86EVAL.Min.WIM
    5,443,684,448 WIN81UX64EVAL.Full.HV.WIM
    5,442,443,606 WIN81UX64EVAL.Full.WIM
    4,723,143,084 WIN81UX64EVAL.Min.HV.WIM
    4,722,734,367 WIN81UX64EVAL.Min.WIM
    4,278,679,489 WIN81UX86EVAL.Full.HV.WIM
    4,277,099,032 WIN81UX86EVAL.Full.WIM
    3,648,938,088 WIN81UX86EVAL.Min.HV.WIM
    3,650,404,303 WIN81UX86EVAL.Min.WIM
    40 File(s) 172,408,546,058 bytes

    Post Processing

    I have scripts to merge similar install wims together to save space. This is similar to what Microsoft does with the Windows Release DVD’s, putting multiple SKU’s in the same *.wim file.


    4,298,516,365 WIN10STPX64.SRV.wim
    4,053,529,276 WIN10TPX64.ENT.wim
    3,138,798,185 WIN10TPX86.ENT.wim
    6,750,236,116 WIN2008R2SP1.wim
    5,292,634,524 WIN2012R2U.wim
    6,785,580,325 WIN7SP1X64EVAL.wim
    5,080,754,294 WIN7SP1X86EVAL.wim
    6,067,072,626 WIN81UX64EVAL.wim
    4,684,309,077 WIN81UX86EVAL.wim
    9 File(s) 46,151,430,788 bytes

    Additionally, I tried out Johan’s Beyond Zip method to shrink files down even more…
    http://www.deploymentresearch.com/Research/tabid/62/EntryId/148/Beyond-Zip-How-to-store-183-GB-of-VMs-in-a-19-GB-file-using-PowerShell.aspx


    23,190,306,816 CapturePackage.vhdx

    From 160GB down to 21.6GB, an savings of about 87% Wow!

    Finally, I have other scripts to convert the *.wim images to *.vhdx files for easy import into Hyper-V or Azure. See my last post on persistalldeviceinstalls

    Uploading

    As a service, I’ve been thinking of uploading my updated/patched images for these Operating Systems (and more) to a public internet file sharing site like my OneDrive for Business account. Rebuilding everything from scratch every Patch Tuesday. One drive for business has 1TB for use, I could share the images, how cool would that be?

    First glitch is that OneDrive for Business still has the 2GB file limitation, so that would require splitting the files up into 2047MB chunks and reassembling later.

    However, my biggest problem right now is my ISP connection. Today, I was averaging about 11.14Mbps upload speed to OneDrive. To upload 42GB of Wim files to OneDrive for Business would take more than 8 hours, which is more time than it took to build the images in the first place. That combined with my ISP’s data caps, makes sharing this from my current office cost prohibitive.

    Customers

    Let me know if you are interested in setting up your own imaging factory environment. I’ve already done this for a large Video Chipset Mfg. And I can customize for your needs.

    KeithGa@DeploymentLive.com

    PersistAllDeviceInstalls in Hyper-V Environments

    I’ve been creating some images for virtual machine environments.

    One of the goals of sysprep is to take all the elements that make an operating system specific to a machine, and make it generic for other machines. However if you know that the hardware is the same, you can tell Sysprep not to strip out the installed device drivers and keep them for the next machine.

    There are two ways to persist the devices and drivers when calling sysprep. If you are using Windows 8 or greater, you can add the /Mode:VM switch to the end of the sysprep call. However if you want the process to work in Windows 7 or Windows Server 2008 R2, you need to put the PersistAllDeviceInstalls element in an unattend.xml file and pass that through to sysprep.

    I created an unattend.xml file and placed it in my MDT Litetouch deployment share under the Tools directory.

    This particular unattend.xml file is crafted to work for both x86 and x64 platforms.

    <?xml version="1.0" encoding="utf-8"?>
    <unattend
      xmlns="urn:schemas-microsoft-com:unattend"
      xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
     <settings pass="generalize">
      <component
        name="Microsoft-Windows-PnpSysprep"
        processorArchitecture="amd64"
        publicKeyToken="31bf3856ad364e35"
        language="neutral"
        versionScope="nonSxS">
       <PersistAllDeviceInstalls>true</PersistAllDeviceInstalls>
      </component>
      <component
        name="Microsoft-Windows-PnpSysprep"
        processorArchitecture="x86"
        publicKeyToken="31bf3856ad364e35"
        language="neutral"
        versionScope="nonSxS">
       <PersistAllDeviceInstalls>true</PersistAllDeviceInstalls>
      </component>
     </settings>
    </unattend>

    Within MDT LiteTouch I then can then set my CustomSettings.ini file to:

    [default]
    OSDAnswerFilePath=%deployroot%\tools\HyperV_Preserve.xml

    Testing

    After building out some of my virtual machines, I decided to run some performance tests against the images that got the PersistAllDeviceInstalls, and those that did not.

    I have a script that will convert a *.wim to a *.vhdx file, and inject a custom unattend.xml file. Import the *.vhdx file into a virtual machine and start up.

    For the image that was given PersistAllDeviceInstalls, it took 1 minute 20 seconds from the start of the virtual machine to the logon prompt.

    For the normal image without PersistAllDeviceInstalls, it took about 3 minutes from the start of the virtual machine to the logon prompt.

    That cut our install time down by almost HALF! Pretty cool!

    -k

    Next up, playing around with the VDI optimization scripts from Jeff and Carl

    http://blogs.technet.com/b/jeff_stokes/archive/2013/04/09/hot-off-the-presses-get-it-now-the-windows-8-vdi-optimization-script-courtesy-of-pfe.aspx

    My Server of Choice for 2014

    I posted some server specs today for one of my servers (yes I have more than one), and it got some attention. Really it’s not my best server.

    My reference server of choice this year is my SuperMicro box:

    supermicro

    * SUPERMICRO SYS-5038D-I Mid-Tower Server
    * 32GB Crucial 240-Pin DDR3 SDRAM ECC Buffered Ram
    * Intel Intel Xeon E3-1270V3 Haswell 3.5GHz
    * Seagate Barracuda ST2000DM001 2TB 7200 RPM
    * (2x) SAMSUNG 840 EVO 250GB SATA Drives

    I load it up with Windows Server 2012 R2 Server (eval) for testing, and it runs like a dream!

    I use 1 SSD drive for the OS, and the other for holding most of my virtual machines.

    Right now I have about 13-15 Virtual Machines up and running with the Windows 8.1 Enterprise Management Suite. No Problem! https://connect.microsoft.com/pocjumpstart/program8614

    VMList

    And it’s only using about 55% of the available memory!

    Whole package should be less than $1500 on Newegg.

    Do you have a better configuration for hosting 10-15 Virtual Machines? let me know!


    Update: Forgot to mention my best server:

    The best server I ever had was at Microsoft when I worked in the extended “System Center” group

    HP Z800
    * 96GB of ECC Ram
    * (2x) Intel Xeon E5640 (if I recall) processors (each with 4 cores)
    * (4x) 300GB Seagate 15k SAS drives ( yes 15000 RPM’s )

    The most obscene part was that I had (TWO) of these machines. I will miss them. :^(

    WP_20130328_001

    Together they would cost about as much as I paid for my last car.