Pipe Dream: Data migration with batch files

So I was working on a problem at work today.  We can only copy 4000 files at a time and I need to copy millions of files.  I grabbed a directory structure with file counts using treesize and dumped that to an excel spreadsheet.  Next, I wrote a program to take all folders that have more than 4000 files in them and separate them into smaller chunks.  The next step was to output batch files with thousands of xcopy commands inside that could run during certain windows on my servers.  this is where I ran into a snag.  What do you do for folders that have more than 4000 files?  We separated them out but I do not have the file names for all the files inside those folders.  We only have the folder name and the file count.  Well, I came up with an idea.  I thought, how about I put variables into the batch file I am programmatically creating and then use a command such as dir to send the files to the copy command.  I typed out some pseudo batch file code and I talked about it.  Here is what I had.
dir [source] @1 | xcopy [source]@1 destination@1
The dir command or some similar command would output all the files in the directory specified as [source] and put them one by one into the variable @1 which would then run xcopy to copy those files using the pipe |.  Well, here comes the name of my blog post.  The idea was good and well received by the team but the commands did not support a syntax like that so I termed the idea my “pipe dream”.

About The Author

Eric Vanderburg

Eric Vanderburg is an author, thought leader, and consultant. He serves as the Vice President of Cybersecurity at TCDI and Vice Chairman of the board at TechMin. He is best known for his insight on cybersecurity, privacy, data protection, and storage. Eric is a continual learner who has earned over 40 technology and security certifications. He has a strong desire to share technology insights with the community. Eric is the author of several books and he frequently writes articles for magazines, journals, and other publications.

Leave a Reply