Monday, November 15, 2010

Ignore Missing Files with tar

I've been experiencing an annoying problem with tar in which it is not creating archives if at least one file in the tar list does not exist. This behavior is totally acceptable and appreciated when I am tarring on demand. However, in this case, I am running a command across several machines each of which will have a different sample of a set of files and I need this done as a cron. And to make matters more complicated the file set involves specific files in a set of directories.
# The novice command
bash$ tar cfz yesterday-logs.tgz \
log-type1/log-type1-2011-11-15.log \
log-type2/log-type2-2011-11-15.log \
log-type3/log-type3-2011-11-15.log \
log-type4/log-type4-2011-11-15.log
This command will work fine if all the directories exist. However, if at least one of the directories or files does not exist, I will experience the following error and an empty tar archive.

tar: log-type3/log-type3-2011-11-15.log: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

I solved this issue by using the wonders bash expansion, sub-shell, and std error redirection
bash$ tar cfz yesterday-logs.tgz $( ls log-type{1,2,3,4}/*-2011-11-15.log} 2>/dev/null )
Fun. I suppose there is most likely a flag to tar that I couldn't find in my scan of the manual entry that completely obviate this complexity, but in the meantime...

6 comments:

  1. So it turns out that tar will ignore missing files by default, so simply piping error output to dev null will do the trick of streaming the tar. Just like I figured: I was overcomplicating things!

    ReplyDelete
    Replies
    1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. big data projects for students But it’s not the amount of data that’s important.Project Center in Chennai

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.

      Spring Training in Chennai

      The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Original idea with bash expansion is still good if using the above in a script because even though tar works with the missing file it will return a non-zero status which could freak out the calling script.

    ReplyDelete
  3. It's great that you described it to me.

    ReplyDelete
  4. Trying to find the silver bullet I chose to use the "tar --ignore-failed-read" option, because I want to create archives even if some files are missing, but at the same time I want to know which files are missing.

    ReplyDelete