Copy-ItemWithProgress

A good friend of mine, Tim, once commented, half jokingly, that his job was to watch Progress Bars. Ha, so true! :^). Most of the work I do involves setting up Operating Systems, and the infrastructure behind it. This means copying a *LOT* of files back and forth. *.ISO, *.Wim, *.vhd, *.vhdx, Drivers, Scripts, whatever. Lately, with most of the control work being done by PowerShell Scripts, and with more of my scripts being wrapped by PowerShell host programs, it was time to re-consider how my large file copies are handled in my PowerShell scripts.

PowerShell Copy

PowerShell has built-in functions for file copy, the copy-item cmdlet with the -recurse switch works quite well, however, for larger files, I would instead recommend RoboCopy.exe which is designed to handle large copies in a structured and *robust* manner.

(I won’t go into the details of why RoboCopy is better than copy-item in PowerShell here in this post. That part is assumed. :^)

Of course the problem with RoboCopy in PowerShell, is that the RoboCopy command was not designed to interface directly with the shell environment, instead it just blindly writes data to the stdout console. However, Copy-item isn’t much better at progress either, although it will copy items, and display the items being copied if the “-verbose” switch is specified, it still does not display PowerShell progress.

I did some work in MDT to help design some capture techniques for console programs. We did modify some of the scripts to capture imagex.exe, DISM.exe, wdsmcast.exe, and USMT and display the progress in a native manner with the SCCM Task Sequencer engine. It involved reading the output and looking for the correct search strings.

mdt progress

RoboCopy

I got inspired this week to re-evaluate robocopy progress after reading a post by Trevor Sullivan, a fellow Microsoft MVP: http://stackoverflow.com/questions/13883404

Wow! The post and associated video is very good. However, there were a few things that I wanted to enhance if I wanted to incorporate into my own processes:

  • The script only supports /mir copy, with no other parameters. Sometimes I need to call with other switches like /xd and /xf. (robocopy has an amazing collection of options available).
  • I though it could be speeded up by launching the 2nd robocopy command immediately after the 1st robocopy command. Rather than waiting for the 1st robocopy to complete.
  • Finally, I also wanted to view the progress of not only the files being copied, but the sub-progress of any large file. This was very important, most of my projects involve copying individual files that often over 1GB in size.

Script

<#
.SYNOPSIS
RoboCopy with PowerShell progress.

.DESCRIPTION
Performs file copy with RoboCopy. Output from RoboCopy is captured,
parsed, and returned as Powershell native status and progress.

.PARAMETER RobocopyArgs
List of arguments passed directly to Robocopy.
Must not conflict with defaults: /ndl /TEE /Bytes /NC /nfl /Log

.OUTPUTS
Returns an object with the status of final copy.
REMINDER: Any error level below 8 can be considered a success by RoboCopy.

.EXAMPLE
C:\PS> .\Copy-ItemWithProgress c:\Src d:\Dest

Copy the contents of the c:\Src directory to a directory d:\Dest
Without the /e or /mir switch, only files from the root of c:\src are copied.

.EXAMPLE
C:\PS> .\Copy-ItemWithProgress '"c:\Src Files"' d:\Dest /mir /xf *.log -Verbose

Copy the contents of the 'c:\Name with Space' directory to a directory d:\Dest
/mir and /XF parameters are passed to robocopy, and script is run verbose

.LINK
https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress

.NOTES
By Keith S. Garner (KeithGa@KeithGa.com) - 6/23/2014
With inspiration by Trevor Sullivan @pcgeek86

#>

[CmdletBinding()]
param(
	[Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
	[string[]] $RobocopyArgs
)

$ScanLog  = [IO.Path]::GetTempFileName()
$RoboLog  = [IO.Path]::GetTempFileName()
$ScanArgs = $RobocopyArgs + "/ndl /TEE /bytes /Log:$ScanLog /nfl /L".Split(" ")
$RoboArgs = $RobocopyArgs + "/ndl /TEE /bytes /Log:$RoboLog /NC".Split(" ")

# Launch Robocopy Processes
write-verbose ("Robocopy Scan:`n" + ($ScanArgs -join " "))
write-verbose ("Robocopy Full:`n" + ($RoboArgs -join " "))
$ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs
$RoboRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $RoboArgs

# Parse Robocopy "Scan" pass
$ScanRun.WaitForExit()
$LogData = get-content $ScanLog
if ($ScanRun.ExitCode -ge 8)
{
	$LogData|out-string|Write-Error
	throw "Robocopy $($ScanRun.ExitCode)"
}
$FileSize = [regex]::Match($LogData[-4],".+:\s+(\d+)\s+(\d+)").Groups[2].Value
write-verbose ("Robocopy Bytes: $FileSize `n" +($LogData -join "`n"))

# Monitor Full RoboCopy
while (!$RoboRun.HasExited)
{
	$LogData = get-content $RoboLog
	$Files = $LogData -match "^\s*(\d+)\s+(\S+)"
    if ($Files -ne $Null )
    {
	    $copied = ($Files[0..($Files.Length-2)] | %{$_.Split("`t")[-2]} | Measure -sum).Sum
	    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
	    {
		    write-progress Copy -ParentID $RoboRun.ID -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
		    $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
	    }
	    else
	    {
		    write-progress Copy -ParentID $RoboRun.ID -Complete
	    }
	    write-progress ROBOCOPY -ID $RoboRun.ID -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1]
    }
}

# Parse full RoboCopy pass results, and cleanup
(get-content $RoboLog)[-11..-2] | out-string | Write-Verbose
[PSCustomObject]@{ ExitCode = $RoboRun.ExitCode }
remove-item $RoboLog, $ScanLog

Code Walkthrough

Parameters

[CmdletBinding()]
param(
	[Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
	[string[]] $RobocopyArgs
)

The script only has one parameter, and that’s the set of arguments passed directly to RoboCopy. I also added the ValueFromRemainingArguments flag so that all arguments passed to command line (other than CommonParameters) would be interpreted as $RobocopyArgs.

Hidden

$ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs

When calling Robocopy.exe, I call the script with the Hidden WindowStyle. This means that the RoboCopy window is not displayed during execution. This is important to me, since I may wish to call RoboCopy from within a non-console window PowerShell Host.

Error Handling

if ($ScanRun.ExitCode -ge 8)
{
	$LogData|out-string|Write-Error
	throw "Robocopy $($ScanRun.ExitCode)"
}

I did add one check into the script to check for obvious errors, for example if you were to call RoboCopy with an incorrect number of parameters, for example one. This would detect the error from the 1st RoboCopy pass, and halt the program with the errors passed to the calling program.

Parsing Logs

$LogData = get-content $RoboLog
$Files = $LogData -match "^\s*(\d+)\s+(\S+)"
   if ($Files -ne $Null )
   {
    $copied = ($Files[0..($Files.Length-2)] | %{$_.Split("`t")[-2]} | Measure -sum).Sum
    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
    {
	    write-progress Copy -ParentID $RoboRun.ID -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
	    $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
    }
    else
    {
	    write-progress Copy -ParentID $RoboRun.ID -Complete
    }
    write-progress ROBOCOPY -ID $RoboRun.ID -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1]
}

I’ll admit that there is some ungraceful code here when parsing the main RoboCopy log file. We are looking for several things here. A) We are parsing for a list of Files copied and their sizes. B) We are also looking for the percentage of the current file being copied.

When parsing the sub-status of large files, I only look for those files large enough to merit having one tenth of a percent displayed, rather than a full percent ( 42.4% would be parsed, however 42% would not).

Testing and Feedback

I did some testing within a console, and within a PowerShell host (in this case the PowerShell ISE).

Host Progress

Console Progress

I have only done some limited testing on my Windows 8.1 Dev box, please let me know if you have any feedback.

Files

Copy-ItemWithProgress.ps1

Advertisements

15 thoughts on “Copy-ItemWithProgress

  1. Since there are only a limited subset of allowed switches, maybe you could use a validateset on $RobocopyArgs to filter out any unwanted switches?

    • Ha, two problems with that, 1. I would hardly call the list of available Robocopy parameters “limited”, there are several dozen, too much effort. 2. I also wanted to allow other parameters not just switches, and they can’t be whitelisted. for example:
      Robocopy /mir c:\test1 c:\test2 *.foo.* /xd Source /xf *.exe *.iso *.vhd *.wim somefile.txt

  2. Hi keith,

    the line write-progress ROBOCOPY -ID $RoboRun.ID -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split(“`t”)[-1] sometimes throw errors when ($Copied/$FileSize*100) is greater than 100 or lower than 0

    Diagg

  3. Hi Keith!

    I was testing this script in my MDT lab today.

    I’m running it as an application in MDT with the following command:

    powershell.exe -executionpolicy bypass -File Copy-ItemWithProgress.ps1 .\Source C:\Data /e -Verbose

    I’m copying a “Source” folder from MDT deployment share where this powershell script is placed. When the applications is run it succesfully copies some of the data, but halts with following displayed in console window:

    cmdlet Write-Progress at command pipeline position 1
    Supply values for the following parameters:
    (Type !? for Help.)
    Status:

    If I hit enter it continues to copy files from subfolders and shows this error:

    Write-Progress : Cannot bind argument to parameter ‘Status’ because it is an em
    pty string.

    It still seems to copy all the data.

    What could be wrong?

    Also is it somehow possible to show the copy progress in MDT task sequencer?

  4. It’s possible that there is a file in your directory with an unusual character or something.

    If you want to dig down and find out, then disable the “remove-Item” at the bottom of the script, and re-run. If you come across a scenario where it fails, please zip me a copy of the “Robocopy Full” log. kg at KeithGa.com

  5. Hi Keith,

    Line 65….
    ($FileSize = [regex]::Match($LogData[-4],”.+:\s+(\d+)\s+(\d+)”).Groups[2].Value)
    …Is causing a null array error on the $LogData var (cannot index into a null array).
    The ScanLog and RoboLog temp files are being created, however have nil content.
    FWIW, I’m running it with no parameters except a source and destination directory and I’m using Windows 7.

    Do you have any ideas please?
    Lyndon

      • Thanks for the reply. Understood, I couldn’t work out why the log files weren’t populating past the file creation. Turns out my path had spaces in the name, and so it wasn’t be parsed correctly within the script even though I was using quotes.

        Regarding the params: Once I worked that out I put my parameters back in. From my old backup solution I had /ETA in, which was causing problems with the script. Once I took that out it started to at least start robocopy, however the screen isn’t updating a progress.

        My new problem is it’s asking for me to supply values for the parameter “Status” during the write-progress line on line 83. I can’t see any reference to this param in the script.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s