{"id":309,"date":"2018-08-22T19:16:53","date_gmt":"2018-08-23T03:16:53","guid":{"rendered":"https:\/\/cloudinsidr.com\/content\/?p=309"},"modified":"2018-09-04T05:31:12","modified_gmt":"2018-09-04T13:31:12","slug":"how-to-rsync-files-and-directories-between-remote-hosts","status":"publish","type":"post","link":"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/","title":{"rendered":"How to use rsync to transfer files and directories between hosts"},"content":{"rendered":"<p>The rsync utility can backup files, synchronize directory trees, and much much more, both on the local machine and between two different\u00a0hosts\u2014via push and pull. Here is how to tame it.<\/p>\n<p><!--more--><\/p>\n<p>This article\u00a0explains:<\/p>\n<ul>\n<li>how to synchronize the contents of two local directories<\/li>\n<li>how to perform a secure rsync via ssh:\n<ul>\n<li>an rsync push from the local host to a remote host<\/li>\n<li>an rsync pull from a remote host to the local host.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>It goes without saying that rsync can have disastrous consequences if performed incorrectly, so whenever in doubt, initiate\u00a0a <strong>&#8211;dry-run<\/strong>\u00a0and always think before you type.<\/p>\n<h2>rsync for starters: synchronizing the contents of two local\u00a0directories<\/h2>\n<p>To synchronize the contents of two directories on your local machine, use the rsync utility like this:<\/p>\n<pre class=\"lang:sh decode:true\">rsync -r source-directory\/ destination-directory<\/pre>\n<p>The trailing slash \/ ensures that the contents of your source-directory would transfer to the destination-directory (in this case) on the same host.<\/p>\n<p>Without the trailing slash, your source-directory would become a sub directory of your destination-directory (in this case, both on the same host).<\/p>\n<p>The -r option\u00a0sets the\u00a0recursive mode. It tells\u00a0rsync to\u00a0traverse subdirectories.<\/p>\n<h2>Remote rsync via ssh<\/h2>\n<p>Synchronizing local directories using rsync is a piece of cake, but\u00a0what if you want to securely rsync files and directories between two separate hosts using ssh with public key authentication over the network? Easy.<\/p>\n<p>There are two ways to accomplish\u00a0this:<\/p>\n<ul>\n<li><strong>a push operation <\/strong>transfers data from the local host to the remote host,<\/li>\n<\/ul>\n<p>and the other way around:<\/p>\n<ul>\n<li><strong>a pull operation<\/strong> transfers data from the remote host to the local host.<\/li>\n<\/ul>\n<h3>rsync push: transfer data from your local host to a remote host<\/h3>\n<p>To initiate an rsync push from the local host to a remote host (or from one remote host to another) via ssh, you can use this command at the prompt of the host that will push its data:<\/p>\n<pre class=\"lang:sh decode:true\">rsync -azhe \"ssh -p 22 -i \/path\/to\/.ssh\/keypair\/private_key_file\" \/local\/path\/to\/source-directory remoteuser@re.mo.te.ip:\/remote\/path\/to\/destination-directory<\/pre>\n<h3>rsync pull: transfer data from a remote host to your local host<\/h3>\n<p>To initiate an rsync pull from a remote\u00a0host to the local\u00a0host, you need to set up ssh with the private key placed locally (preferably in $HOME\/.ssh of the active local user) and the corresponding public key located on the remote computer (in a separate line inside the file authorized_keys in in $HOME\/.ssh of the remote user).<\/p>\n<p>To set up the keys, follow steps 1 through 5 in this tutorial: <a href=\"https:\/\/cloudinsidr.com\/content\/how-to-set-up-a-connection-between-two-hosts-using-authentication-based-on-key-pairs-for-remote-access\/\" target=\"_blank\" rel=\"noopener\">How to Set Up a Connection between Two Hosts Using Authentication Based on Key Pairs for Remote Access via ssh, rsync<\/a>.<\/p>\n<p>Once you have verified that you can establish an ssh connection,\u00a0you can now use this command:<\/p>\n<pre>rsync -azhe \"ssh -p 22 -i \/path\/to\/.ssh\/keypair\/private_key_file\" --progress remoteuser@&lt;ip-address-or-domain&gt;:\/path\/to\/items\/on\/the\/REMOTE\/HOST\/OBJECT \/path\/to\/items\/on\/the\/LOCAL\/HOST\/DESTINATION\/<\/pre>\n<p>The -p flag specifies\u00a0the ssh port number on the remote host to connect to, so if your remote ssh demon\u00a0listens on a different port number,\u00a0you need to replace the default 22 with that\u00a0port number.\u00a0The parent directory on the target machine (in the above example DESTINATION) must already exist. OBJECT can be several files or directories (several, when using wildcards).<\/p>\n<h2>User mapping in rsync<\/h2>\n<p><a href=\"https:\/\/rsync.samba.org\/ftp\/rsync\/src\/rsync-3.1.0-NEWS\" target=\"_blank\" rel=\"noopener\">rsync version 3.1.0<\/a>\u00a0introduced user mapping with the &#8211;usermap and &#8211;groupmap options.\u00a0This allows you to specify ownership of files on the remote system like this:<\/p>\n<pre class=\"lang:sh decode:true \">--usermap=originowner:destinationowner\r\n--groupmap=origingroup:destinationgroup<\/pre>\n<p>Since option -a will cause rsync to preserve ownership, you may want\u00a0to puzzle its functionality together by setting individual flags (see above section about\u00a0useful rsync options).<\/p>\n<h2>rsync in the cloud with systems that don&#8217;t support\u00a0root login<\/h2>\n<p>Many AMIs (Amazon Machine images) on AWS are configured in a way that disallows remote authentication as root (even though they may contain a\u00a0public key in root&#8217;s authorized_keys file). The administrator connects as an unauthorized user, then enters\u00a0<strong><span class=\"lang:sh decode:true crayon-inline\">sudo su<\/span><\/strong>\u00a0or <strong><span class=\"lang:sh decode:true crayon-inline\">sudo -i<\/span><\/strong>\u00a0 to acquire root privileges. This may be a good practice, but it&#8217;s counterproductive when you want to use <strong>rsync<\/strong>. The utility\u00a0may not be able to write to a directory owned by root:root unless root initiates the connection. So what&#8217;s the fix?<\/p>\n<p>The simplest way to work around the problem is by activating root login in<\/p>\n<pre class=\"lang:sh decode:true \">\/etc\/ssh\/sshd_config<\/pre>\n<p>by using the parameter\u00a0<span class=\"lang:sh decode:true crayon-inline\">PermitRootLogin<\/span>.<\/p>\n<p>This parameter can take one of four values:<\/p>\n<ul>\n<li><strong>PermitRootLogin without-password<\/strong>: this disables password authentication for root, allowing other authentication methods (such as keys);<\/li>\n<li><strong>PermitRootLogin forced-commands-only<\/strong>: this allows root login with public key authentication, but only if the &#8216;command&#8217; option has been specified; no other authentication methods are allowed;<\/li>\n<li><strong>PermitRootLogin no<\/strong>: this setting prohibits login as root regardless of the authentication method used;<\/li>\n<li><strong>PermitRootLogin<del> yes<\/del><\/strong>: allows unsafe login as root (you don&#8217;t want this one).<\/li>\n<\/ul>\n<p>Set this option as follows:<\/p>\n<pre class=\"lang:sh decode:true\">PermitRootLogin without-password<\/pre>\n<p>and append\u00a0your\u00a0<strong>public key<\/strong> into the authorized_keys file that is located\u00a0in the $HOME\/.ssh folder that belongs to root. For more details on how to create and install\u00a0your key pairs\u00a0read\u00a0<a href=\"https:\/\/cloudinsidr.com\/content\/how-to-set-up-a-connection-between-two-hosts-using-authentication-based-on-key-pairs-for-remote-access\/\">How to Set Up a SSH Connection Using Authentication Based on Key Pairs<\/a>.<\/p>\n<p>With\u00a0\u00a0PermitRootLogin without-password in place, SSH will still ask the root user for a password, even though\u00a0no password will work. This behavior is rather unnerving (security by obscurity?). To suppress this behavior, add\u00a0these two lines to \/etc\/ssh\/sshd_config:<\/p>\n<pre class=\"lang:sh decode:true\">Match User root\r\nPasswordAuthentication no<\/pre>\n<p>If you would like to prohibit password authentication for all users on the system, this will do the trick:<\/p>\n<pre class=\"lang:sh decode:true \">PasswordAuthentication no<\/pre>\n<h2>Useful rsync options<\/h2>\n<p>Some useful rsync options include:<\/p>\n<ul>\n<li><strong>a<\/strong> or\u00a0<strong>&#8211;archive<\/strong>: activates the archive mode; this mode tells rsync to:\n<ul>\n<li>traverse directories\u00a0recursively (implies option -r),<\/li>\n<li>preserve:\n<ul>\n<li>symbolic links (-l or &#8211;links) and<\/li>\n<li>other special and device files (-D or &#8211;devices and &#8211;specials); with additional options for\u00a0restricions on link validity and the like,<\/li>\n<\/ul>\n<\/li>\n<li>transfer:\n<ul>\n<li>permissions (implied option -p or &#8211;perms),<\/li>\n<li>user and group ownerships (-o for &#8211;owner and -g for &#8211;group), and<\/li>\n<li>timestamps (-t or &#8211;times),<\/li>\n<\/ul>\n<\/li>\n<li>but does not imply\u00a0the -H option for hard links (this option can be set separately);<\/li>\n<\/ul>\n<\/li>\n<li><strong>e<\/strong> specifies the remote shell to use (see section on rsync via ssh);<\/li>\n<li><strong>h<\/strong> or &#8211;human-readable: outputs file sizes\u00a0in a human-readable format with units of K for kilobytes, M for megabytes, and G for gigabytes; if specified twice (-hh), the units are powers of 1024 instead of the default 1000;<\/li>\n<li><strong>z<\/strong> or <strong>&#8211;compress<\/strong> enables compression;<\/li>\n<li><strong>&#8211;progress<\/strong> outputs a line-by-line activity report (implies -v for\u00a0&#8211;verbose, an option that in itself can be very useful when\u00a0troubleshooting\u00a0connection problems);<\/li>\n<li><strong>R<\/strong> replicates the complete directory tree with an absolute path instead of a relative one (\/the\/complete\/original\/path\/to\/object becomes \/destination\/path\/the\/complete\/original\/path\/to\/object);<\/li>\n<li><strong>n<\/strong> or <strong>&#8211;dry-run<\/strong>\u00a0allows for a risk-free\u00a0test\u00a0run of rsync without actually copying anything; remember to ask for verbose output using either <strong>-v<\/strong>, <strong>&#8211;verbose<\/strong> <strong>or &#8211;progress<\/strong>.<\/li>\n<\/ul>\n<p>rsync supports plenty of other useful options that make for rather sophisticated methods\u00a0of operation. For additional\u00a0inspiration, you can always refer to the manual:<\/p>\n<pre class=\"lang:sh decode:true\">man rsync<\/pre>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The rsync utility can backup files, synchronize directory trees, and much much more, both on the local machine and between two different\u00a0hosts\u2014via push and pull. Here is how to tame it.<\/p>\n","protected":false},"author":1,"featured_media":76,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[16,17,33],"tags":[76,58,74,73,75,19],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to use rsync to transfer files and directories between hosts - CloudInsidr<\/title>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to use rsync to transfer files and directories between hosts - CloudInsidr\" \/>\n<meta property=\"og:description\" content=\"The rsync utility can backup files, synchronize directory trees, and much much more, both on the local machine and between two different\u00a0hosts\u2014via push and pull. Here is how to tame it.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/\" \/>\n<meta property=\"og:site_name\" content=\"CloudInsidr\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-23T03:16:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-09-04T13:31:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.cloudinsidr.com\/content\/wp-content\/uploads\/2015\/11\/cloudinsidr_logo_900px-wide.png\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"326\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#website\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/\",\"name\":\"CloudInsidr\",\"description\":\"Cyber security, infotech\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.cloudinsidr.com\/content\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/wp-content\/uploads\/2015\/11\/cloudinsidr_logo_900px-wide.png\",\"width\":900,\"height\":326,\"caption\":\"cloudinsidr.com logo (900px wide)\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/#webpage\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/\",\"name\":\"How to use rsync to transfer files and directories between hosts - CloudInsidr\",\"isPartOf\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/#primaryimage\"},\"datePublished\":\"2018-08-23T03:16:53+00:00\",\"dateModified\":\"2018-09-04T13:31:12+00:00\",\"author\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#\/schema\/person\/dd6ee9cb21cf05763fd7cff3d6f11b2b\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.cloudinsidr.com\/content\/how-to-rsync-files-and-directories-between-remote-hosts\/\"]}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#\/schema\/person\/dd6ee9cb21cf05763fd7cff3d6f11b2b\",\"name\":\"Cloud Insidr\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8b2fa1415b3d573b97d818b8f8f83b7c?s=96&d=mm&r=g\",\"caption\":\"Cloud Insidr\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/309"}],"collection":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/comments?post=309"}],"version-history":[{"count":29,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/309\/revisions"}],"predecessor-version":[{"id":2356,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/309\/revisions\/2356"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/media\/76"}],"wp:attachment":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/media?parent=309"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/categories?post=309"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/tags?post=309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}