Copy-ItemWithProgress

A good friend of mine, Tim, once commented, half jokingly, that his job was to watch Progress Bars. Ha, so true! :^). Most of the work I do involves setting up Operating Systems, and the infrastructure behind it. This means copying a *LOT* of files back and forth. *.ISO, *.Wim, *.vhd, *.vhdx, Drivers, Scripts, whatever. Lately, with most of the control work being done by PowerShell Scripts, and with more of my scripts being wrapped by PowerShell host programs, it was time to re-consider how my large file copies are handled in my PowerShell scripts.

PowerShell Copy

PowerShell has built-in functions for file copy, the copy-item cmdlet with the -recurse switch works quite well, however, for larger files, I would instead recommend RoboCopy.exe which is designed to handle large copies in a structured and *robust* manner.

(I won’t go into the details of why RoboCopy is better than copy-item in PowerShell here in this post. That part is assumed. :^)

Of course the problem with RoboCopy in PowerShell, is that the RoboCopy command was not designed to interface directly with the shell environment, instead it just blindly writes data to the stdout console. However, Copy-item isn’t much better at progress either, although it will copy items, and display the items being copied if the “-verbose” switch is specified, it still does not display PowerShell progress.

I did some work in MDT to help design some capture techniques for console programs. We did modify some of the scripts to capture imagex.exe, DISM.exe, wdsmcast.exe, and USMT and display the progress in a native manner with the SCCM Task Sequencer engine. It involved reading the output and looking for the correct search strings.

RoboCopy

I got inspired this week to re-evaluate robocopy progress after reading a post by Trevor Sullivan, a fellow Microsoft MVP: http://stackoverflow.com/questions/13883404

Wow! The post and associated video is very good. However, there were a few things that I wanted to enhance if I wanted to incorporate into my own processes:

• The script only supports /mir copy, with no other parameters. Sometimes I need to call with other switches like /xd and /xf. (robocopy has an amazing collection of options available).
• I though it could be speeded up by launching the 2nd robocopy command immediately after the 1st robocopy command. Rather than waiting for the 1st robocopy to complete.
• Finally, I also wanted to view the progress of not only the files being copied, but the sub-progress of any large file. This was very important, most of my projects involve copying individual files that often over 1GB in size.

Script

<#
.SYNOPSIS
RoboCopy with PowerShell progress.

.DESCRIPTION
Performs file copy with RoboCopy. Output from RoboCopy is captured,
parsed, and returned as Powershell native status and progress.

.PARAMETER RobocopyArgs
List of arguments passed directly to Robocopy.
Must not conflict with defaults: /ndl /TEE /Bytes /NC /nfl /Log

.OUTPUTS
Returns an object with the status of final copy.
REMINDER: Any error level below 8 can be considered a success by RoboCopy.

.EXAMPLE
C:\PS> .\Copy-ItemWithProgress c:\Src d:\Dest

Copy the contents of the c:\Src directory to a directory d:\Dest
Without the /e or /mir switch, only files from the root of c:\src are copied.

.EXAMPLE
C:\PS> .\Copy-ItemWithProgress '"c:\Src Files"' d:\Dest /mir /xf *.log -Verbose

Copy the contents of the 'c:\Name with Space' directory to a directory d:\Dest
/mir and /XF parameters are passed to robocopy, and script is run verbose

.LINK
https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress

.NOTES
By Keith S. Garner (KeithGa@KeithGa.com) - 6/23/2014
With inspiration by Trevor Sullivan @pcgeek86

#>

[CmdletBinding()]
param(
	[Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
	[string[]] $RobocopyArgs
)

$ScanLog  = [IO.Path]::GetTempFileName()
$RoboLog  = [IO.Path]::GetTempFileName()
$ScanArgs = $RobocopyArgs + "/ndl /TEE /bytes /Log:$ScanLog /nfl /L".Split(" ")
$RoboArgs = $RobocopyArgs + "/ndl /TEE /bytes /Log:$RoboLog /NC".Split(" ")

# Launch Robocopy Processes
write-verbose ("Robocopy Scan:`n" + ($ScanArgs -join " "))
write-verbose ("Robocopy Full:`n" + ($RoboArgs -join " "))
$ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs
$RoboRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $RoboArgs

# Parse Robocopy "Scan" pass
$ScanRun.WaitForExit()
$LogData = get-content $ScanLog
if ($ScanRun.ExitCode -ge 8)
{
	$LogData|out-string|Write-Error
	throw "Robocopy $($ScanRun.ExitCode)"
}
$FileSize = [regex]::Match($LogData[-4],".+:\s+(\d+)\s+(\d+)").Groups[2].Value
write-verbose ("Robocopy Bytes: $FileSize `n" +($LogData -join "`n"))

# Monitor Full RoboCopy
while (!$RoboRun.HasExited)
{
	$LogData = get-content $RoboLog
	$Files = $LogData -match "^\s*(\d+)\s+(\S+)"
    if ($Files -ne $Null )
    {
	    $copied = ($Files[0..($Files.Length-2)] | %{$_.Split("`t")[-2]} | Measure -sum).Sum
	    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
	    {
		    write-progress Copy -ParentID $RoboRun.ID -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
		    $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
	    }
	    else
	    {
		    write-progress Copy -ParentID $RoboRun.ID -Complete
	    }
	    write-progress ROBOCOPY -ID $RoboRun.ID -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1]
    }
}

# Parse full RoboCopy pass results, and cleanup
(get-content $RoboLog)[-11..-2] | out-string | Write-Verbose
[PSCustomObject]@{ ExitCode = $RoboRun.ExitCode }
remove-item $RoboLog, $ScanLog

Code Walkthrough

Parameters

[CmdletBinding()]
param(
	[Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
	[string[]] $RobocopyArgs
)

The script only has one parameter, and that’s the set of arguments passed directly to RoboCopy. I also added the ValueFromRemainingArguments flag so that all arguments passed to command line (other than CommonParameters) would be interpreted as $RobocopyArgs.

Hidden

$ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs

When calling Robocopy.exe, I call the script with the Hidden WindowStyle. This means that the RoboCopy window is not displayed during execution. This is important to me, since I may wish to call RoboCopy from within a non-console window PowerShell Host.

Error Handling

if ($ScanRun.ExitCode -ge 8)
{
	$LogData|out-string|Write-Error
	throw "Robocopy $($ScanRun.ExitCode)"
}

I did add one check into the script to check for obvious errors, for example if you were to call RoboCopy with an incorrect number of parameters, for example one. This would detect the error from the 1st RoboCopy pass, and halt the program with the errors passed to the calling program.

Parsing Logs

$LogData = get-content $RoboLog
$Files = $LogData -match "^\s*(\d+)\s+(\S+)"
   if ($Files -ne $Null )
   {
    $copied = ($Files[0..($Files.Length-2)] | %{$_.Split("`t")[-2]} | Measure -sum).Sum
    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
    {
	    write-progress Copy -ParentID $RoboRun.ID -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
	    $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
    }
    else
    {
	    write-progress Copy -ParentID $RoboRun.ID -Complete
    }
    write-progress ROBOCOPY -ID $RoboRun.ID -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1]
}

I’ll admit that there is some ungraceful code here when parsing the main RoboCopy log file. We are looking for several things here. A) We are parsing for a list of Files copied and their sizes. B) We are also looking for the percentage of the current file being copied.

When parsing the sub-status of large files, I only look for those files large enough to merit having one tenth of a percent displayed, rather than a full percent ( 42.4% would be parsed, however 42% would not).

Testing and Feedback

I did some testing within a console, and within a PowerShell host (in this case the PowerShell ISE).

I have only done some limited testing on my Windows 8.1 Dev box, please let me know if you have any feedback.

Files

Copy-ItemWithProgress.ps1

Advertisement