One of the first Octopus scripts I wrote was one to copy files and folders — a pretty basic deployment activity. But periodically I found that resources were getting locked. I noticed that this was happening mostly when I stopped a Windows service prior to deleting and redeploying it.
That makes sense. Typically when you issue a Stop command to a service, you signal its worker thread or threads, and then wait for them to check the Event, note its changed status, and work out how they’re going to clean up their resources. So, depending on how you’ve implemented things, your service’s main thread may officially stop somewhat ahead of its worker threads.
Copying files with repeated polling
This shouldn’t be a big deal, but Powershell can get very fussy when dealing with locked resources. For example, call Copy-Item with the -Force flag when the destination is locked. Bam! You’re in the exception handler. Call Test-Path on a non-existent directory? Same thing. But I soon got round these hurdles, and had a basic script that would just spin for a bit, retry the copy, and finally throw an error if it just couldn’t update the file. It went something like this:
for ($i = 1; $i - le $Timeout -
and(Test - Path - Path $DestinationDir -
ErrorAction
SilentlyContinue);
$i++) {
Write -
Host
"Attempting to remove folder $DestinationDir ($i of $Timeout)" Remove -
Item - Path $DestinationDir - Recurse - Force -
ErrorAction SilentlyContinue Start - Sleep - Milliseconds 1000
}
if (Test - Path - Path $DestinationDir - ErrorAction SilentlyContinue) {
throw "Unable to clean $DestinationDir; aborting"
}Copying files and closing handles
But this wasn’t the whole story. The script worked most of the time, but every now and then files or folders still got locked. I logged on the server in question, checked with Task manager, and soon saw what was going on. I’d got the folder open in a command shell, or else the file open in Notepad ++. “That’s easy,” I thought. “If I can’t copy a file, I’ll check for open handles, and just close them.” Here’s the full script:
function Close - FileHandles {
param(
[parameter(mandatory = $True,
HelpMessage = 'Full or partial file path')][string] $FilePath)
Write -
Host "Searching for locks on path: $FilePath"
gps |
Where - Object { $_.Path -match $FilePath.Replace("\", "\\") } |% `
{
Write - Host "Closing process $_.Name with path: $_.Path" Stop - Process -
Id $_.Id - Force
}
}
function Copy - FilesAndFolders {
param(
[parameter(
mandatory = $True,
HelpMessage =
'Source directory from which to copy')][string] $SourceDir,
[parameter(
mandatory = $True,
HelpMessage =
'Destination directory to which to copy')][string] $DestinationDir,
[parameter(
mandatory = $True,
HelpMessage =
'Clean destination directory before copying')][bool] $CleanDestinationDir =
$False,
[parameter(mandatory = $False, HelpMessage = 'Timeout')][int] $Timeout =
20)
#Ensure script fails on any errors
$ErrorActionPreference = "Stop"
if ($CleanDestinationDir - eq $True) {
#First, try to remove the folder without explicitly dropping any locks; leave a
reasonable
#time for services to stop and threads to end
for ($j = 0; $j - le 2; $j++) {
Write - Host
"Attempting to delete $DestinationDir, since CleanDestinationDir was
specified "
for ($i = 1; $i - le $Timeout -
and(Test - Path - Path $DestinationDir -
ErrorAction
SilentlyContinue);
$i++) {
Write -
Host
"Attempting to remove folder $DestinationDir ($i of $Timeout)" Remove -
Item - Path $DestinationDir - Recurse - Force -
ErrorAction SilentlyContinue Start - Sleep - Milliseconds 1000
}
if (Test - Path - Path $DestinationDir - ErrorAction SilentlyContinue) {
if ($j - eq 0) {
Write -
Host
"Folder is still locked; dropping locks and retrying" Close -
FileHandles - FilePath $DestinationDir
} else {
throw "Unable to clean $DestinationDir; aborting"
}
}
}
}
Write -
Host "Checking whether $DestinationDir directory exists"
if (!(Test - Path $DestinationDir - ErrorAction SilentlyContinue)) {
#Create the destination folder
Write - Host "$DestinationDir does not exist; creating" New -
Item - ItemType directory - Path $DestinationDir
}
#Copy the folder
Write -
Host "Copying $SourceDir to $DestinationDir" Get - ChildItem -
Path $SourceDir |
% ` {
try {
Write - Host "Copying: $_" Copy - Item $_.fullname "$DestinationDir" -
Recurse - Force
} catch {
Write -
Host
"Destination file appears to be locked; attempting to drop locks on
folder "
Close -
FileHandles - FilePath $DestinationDir
}
}
}Now this works a lot better. For example, it can now find and drop locks held by applications like Notepad ++. But it still isn’t the whole story. Try locking a folder by leaving a command shell open, and then try the script. It doesn’t seem to twig that cmd.exe is the culprit, and you can see why when you look through the detail supplied by Get-Process. The command shell is actually executing in the Windows system directory, so what you really wanted to do was to kill any process whose current working directory was within the folder structure you were trying to copy.
This seems like a pretty obvious thing to want to do. It’s actually rather difficult, and the only way I was able to find was via the Windows API, using CreateRemoteThread to inject a call to GetCurrentDirectory into the external process.
Visions of P/Invoking the Windows API across process boundaries via .NET, wrapped in Powershell, correctly managing my memory allocations and deallocations, pointers and pointer sizes … it really didn’t appeal. And then I remembered that this is exactly what SysInternal’s “handle.exe” program does. All I’d need to do was shell out to it, like this.
The Sledgehammer approach
The script was definitely improving, but now I found myself wondering how I was going to get handle.exe onto my target boxes in the first place. I wasn’t using a NuGet package, and really what I wanted to do was to store all my deployment functionality in a few Powershell script libraries. Then it occurred to me I could do something really perverse — I could embed handle.exe (or any other resource) in my Powershell script with Base 64 encoding, and then extract it to a temporary file on demand. I grabbed a couple of conversion functions from here.
I then invoked ConvertTo-EncodedText to get me an encoded version of handle.exe, and I pasted its contents into the method GetBLOB-HandleEx below. When you invoke Copy-FilesAndFolders, it first tries to copy the old fashioned way. If that fails, it checks for handle.exe in the “Tools” directory, invokes it if it’s there, or extracts it if it isn’t. This was an optimisation, since it takes quite a while to extract a large embedded resource, so I opted only to do this if it was necessary.
function GetBLOB -
HandleEx() {
return "TVqQAAMAAAAEAAAA// ... and so on for another 500Kb ... AAAAAA"
}
function InstallOnAccess
- HandleEx($path) {
Write - Host "Checking for handle.exe at $path"
if (!(Test - Path - Path $path)) {
Write - Host "Unpacking and installing" ConvertFrom - EncodedText -
InputData(GetBLOB - HandleEx) - SaveTo "$path" Write - Host "Done"
}
}
function Drop - LocksOnFolder($pathToFree, $toolsDir) {
#Create Tools directory, if it doesn't already exist
Write - Host "Dropping locks on folder: $pathToFree"
if (!(Test - Path -
Path $toolsDir)) { Write -
Host "Creating ToolsDirectory at $toolsDir" New -
Item - ItemType directory - Path $pathToFree }
#Extract Handle.exe, if we don't already have it
$handleExPath =
"$toolsDir\handle.exe" InstallOnAccess - HandleEx($handleExPath)
$results =
&$handleExPath / accepteula $handles = $results |
where { $_ -match "File" -and $_ -match $pathToFree.Replace("\", "\\") }
$handles = $handles |% `
{
$handle = $_.Substring(0, $_.IndexOf(':')).Trim();
Write-Host "Closing handle: $handle"
& handle -c $handle
}
}
function Copy - FilesAndFolders {
param(
[parameter(
mandatory = $True,
HelpMessage =
'Source directory from which to copy')][string] $SourceDir,
[parameter(
mandatory = $True,
HelpMessage =
'Destination directory to which to copy')][string] $DestinationDir,
[parameter(
mandatory = $True,
HelpMessage =
'Clean destination directory before copying')][bool] $CleanDestinationDir =
$False,
[parameter(mandatory = $False,
HelpMessage = 'Tools directory')][string] $ToolsDir)
#Ensure script fails on any errors
$ErrorActionPreference = "Stop"
if ($CleanDestinationDir - eq $True) {
#Remove the destination folder if it already exists
Write -
Host
"Attempting to delete $DestinationDir, since CleanDestinationDir was
specified "
Remove -
Item - Path $DestinationDir - Recurse - Force -
ErrorAction SilentlyContinue
#If we couldn't remove it, something has a lock on it
if (Test - Path - Path $DestinationDir -
ErrorAction SilentlyContinue) {
Drop - LocksOnFolder - pathToFree $DestinationDir - toolsDir $ToolsDir
}
}
Write -
Host "Checking whether $DestinationDir directory exists"
if (!(Test - Path $DestinationDir)) {
#Create the destination folder
Write - Host "$DestinationDir does not exist; creating" New - Item -
ItemType directory - Path $DestinationDir
}
#Copy the folder
Write -
Host "Copying $SourceDir to $DestinationDir" $failing = $True
while ($failing) {
$failing = $False
Get -
ChildItem - Path $SourceDir |
% ` {
try {
Copy - Item $_.fullname "$DestinationDir" - Recurse - Force -
ErrorAction Stop
} catch[System.Exception] {
Write - Host "File handle open in destination for $_" $failing =
$True Drop - LocksOnFolder - pathToFree $DestinationDir -
toolsDir $ToolsDir
}
}
}
}Ultimately, I found this script was overkill, and the weight of a large embedded binary object took its toll when I tried working on the script in Powershell ISE. but I’m including the code to show how my thinking evolved, and possibly to give you some ideas of your own about novel payloads in your scripts!
Posted on Thursday, January 8, 2015 8:55 PM | Back to top
