Streaming an archive from a server over SSH and extracting it on the fly

I’d found this command a while ago now but had completely forgotten about it until I stumbled upon it yesterday. So, if you had one or more tar files on a server and wanted to get them all down and then extract them somewhere to do your work, the normal way to achieve that would be:

scp awesomeuser@awesomeserver:/tmp/mytarfile.tar /storage/

The problem with this is that its a two step action. 1) You download the tar file(s). 2) You extract the downloaded tar file(s). While this is “functional”, here’s an awesome way to do it:

curl -u awesomeuser: --key /home/awesomeuser/.ssh/id_rsa --pubkey /home/awesomeuser/.ssh/id_rsa.pub scp://awesomeserver/tmp/mytarfile.tar | tar xf -

Lets break that down. We’re using the curl command to download the file using awesomeuser as the username (indicated by -u awesomeuser). For authentication, we are using our private and public keys as indicated by --key /home/awesomeuser/.ssh/id_rsa and --pubkey /home/awesomeuser/.ssh/id_rsa.pub flags. We’re then asking the curl command to use the scp protocol for accessing our server (as indicated by scp://awesomeserver) and are then giving it the location of the tar file that we want to download (as indicated by /tmp/mytarfile.tar). Because curl command will output whatever it gets to standard out stream (STDOUT), we’re leveraging this capability by redirecting STDOUT to a pipe which is then fed into the tar command (as indicated by | tar xf -).

In case you have a tar.gz file or a tgz file, use:

curl -u awesomeuser: --key /home/awesomeuser/.ssh/id_rsa --pubkey /home/awesomeuser/.ssh/id_rsa.pub scp://awesomeserver/tmp/mytarfile.tar.gz | gunzip -c - | tar xf -

Same principle except before the final pipe of tar xf -, we’re squeezing gunzip -c in between to g-unzip the file first before untarring it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.