So I was working on a problem at work today.  We can only copy 4000 files at a time and I need to copy millions of files.  I grabbed a directory structure with file counts using treesize and dumped that to an excel spreadsheet.  Next I wrote a program to take all folders that have more than 4000 files in them and separate them into smaller chunks.  The next step was to output batch files with thousands of xcopy commands inside that could run during certain windows on my servers.  this is where I ran into a snag.  What do you do for folders that have more than 4000 files?  We separated them out but I do not have the file names for all the files inside those folders.  We only have the folder name and the file count.  Well, I came up with an idea.  I thought, how about I put variables into the batch file I are programmatically creating and then use a command such as dir to send the files to the copy command.  I typed out some pseudo batch file code and I talked about it.  Here is what I had.

dir [source] @1 | xcopy [source]@1 [email protected]

The dir command or some similar command would output all the files in the directory specified as [source] and put them one by one into the variable @1 which would then run xcopy to copy those files using the pipe |.┬á Well, here comes the name of my blog post.┬á The idea was good and well received by the team but the commands did not support a syntax like that so I termed the idea my “pipe dream”.