Easily Synchronise Folders Locally and Remotely

Folder synchronisation is one of the most frequent and crucial operations in file management on modern operating systems. Whether copying, updating or maintaining two identical folders on multiple devices or servers, data synchronisation is essential to ensure file integrity and availability. In this article, we will explore how to synchronise two folders locally or remotely on all popular operating systems, such as Windows, Linux and macOS, using powerful tools and versatile commands.

1. Local vs. Remote Synchronisation: What is the Difference?

• Local Synchronisation: This is the copying of files or folders between two locations on the same computer or device. For example, copying data from one folder to another on the same hard drive or on a different partition.
• Remote Synchronisation: This operation involves copying files between two devices that may be on different networks or over long geographical distances. It is often used for backup, content distribution and server management. Data can be synchronised using network protocols such as SSH, FTP or SFTP.

2. Synchronisation Tools for Different Operating Systems

There are several tools that allow you to synchronise files and folders, both locally and remotely, easily and efficiently. In the following, we will explore the main commands for various operating systems.

3. Synchronising on Linux and macOS with rsync

One of the most powerful and flexible tools for synchronising folders on Linux and macOS is rsync. This tool allows files to be copied and synchronised incrementally, which means that only changed files will be transferred, saving time and resources.

Basic syntax of rsync:

rsync [options] <origin> <destination>.

Where:

• <origin> is the source folder or file to be copied.
• <destination> is the destination folder.
• [options] are the parameters to define the synchronisation behaviour.

Main rsync options :

• -a, --archive: Enables archive mode, which recursively copies folders while maintaining permissions, timestamps, symbolic links, devices, and owners. This is the most common option for backup operations.
  
  • Example: rsync -a /source/ /destination/
• -r, --recursive: Copies subdirectories recursively.
  
  • Example: rsync -r /source/ /destination/
• -v, --verbose: Shows detailed information about the copy process.
  
  • Example: rsync -v /source/ /destination/
• --progress: Shows the progress of the file copy (including transfer rate and estimated completion time).
  
  • Example: rsync --progress /source/ /destination/
• --delete: Deletes files in the destination that are no longer present in the source. Useful for creating a perfect synchronisation (mirror).
  
  • Example: rsync --delete /source/ /destination/
• --exclude=pattern: Excludes files or folders that match the specified pattern.
  
  • Example: rsync --exclude=‘*.tmp’ /source/ /destination/
• --include=pattern: Only include files that match the specified pattern.
  
  • Example: rsync --include=‘*.jpg’ --exclude=‘*’ /source/ /destination/
• -z, --compress: Enables file compression during transfer, reducing bandwidth usage.
  
  • Example: rsync -z /source/ /destination/
• -e, --rsh=COMMAND: Specifies the remote command for the connection (such as SSH).
  
  • Example: rsync -e ssh /source/ user@remote:/destination/
• --dry-run: Performs a simulation of synchronisation without actually transferring files.
  
  • Example: rsync --dry-run /source/ /destination/

Full Remote Synchronisation Example with rsync:

rsync -avz --delete --progress --exclude=‘*.log’ /local/folder/ user@remote:/remote/folder/

This command:

• Synchronise in archive mode (-a),
• Show progress (--progress),
• Compress data during transfer (-z),
• Deletes files in the destination that are no longer in the source (--delete),
• Excludes files with the extension .log (--exclude=‘*.log’).

Final Full Remote Synchronisation Example enhanced even for bad connections with rsync:

rsync -avzHc -e "ssh -p 22" --partial --inplace --timeout=60 --bwlimit=1000 --compress-level=1 --ignore-existing --delete --dry-run --exclude=/proc --exclude=/dev --exclude=/sys --exclude=/tmp root@server.example.com:/ /mnt/backup/

This command:

• -H: It does what the previous command does but additionally preserves hardlink.
• -c: skip based on checksum, not mod-time & size.
• --partial: Keeps partially copied files if the connection is interrupted.
• --inplace: Writes directly to the destination file.
• --timeout=60: Timeout 60 seconds for the connection.
• --bwlimit=1000: Limits bandwidth to 1 MB/s.
• --compress-level=1: Reduces the compression level to save bandwidth.
• -e: Specify the port.
• --dry-run: Simulate the rsync to test. You can remove if the test is successful. Do it always because it's an excellent idea.
• --exclude: Exclude some particular path to avoid errors.

4. Synchronisation on Windows with robocopy

On Windows, the main tool for synchronising files and folders is robocopy (Robust File Copy). This command, introduced in Windows Vista, is much more powerful than the traditional xcopy command.

Basic syntax of robocopy:

robocopy <origin> <destination> [file(s)_to_copy] [options].

robocopy main options :

• /S: Copies all subdirectories, but not empty ones.
  
  • Example: robocopy C:\source D:\destination /S
• /E: Copies all subdirectories, including empty ones.
  
  • Example: robocopy C:\source D:\destination /E
• /MIR: Creates a mirror of the source into the destination, deleting files that are no longer present in the source.
  
  • Example: robocopy C:\source D:\destination /MIR
• /MOVE: Moves files and subdirectories from the source to the destination, deleting them from the source.
  
  • Example: robocopy C:\source D:\destination /MOVE
• /COPY:DAT: Copies data, attributes and timestamps. The letters specify the properties to be copied (e.g. D for data, A for attributes, T for timestamps).
  
  • Example: robocopy C:\source D:\destination /COPY:DAT
• /Z: Allows copying in robocopy mode (useful in case of interruptions).
  
  • Example: robocopy C:\source D:\destination /Z
• /R:n: Specifies the number of retries to retry the file transfer in case of an error (default is 1 million).
  
  • Example: robocopy C:\source D:\destination /R:3
• /W:n: Sets the number of seconds to wait between copy attempts (default is 30 seconds).
  
  • Example: robocopy C:\source D:\destination /W:10
• /LOG:file: Records the output of the command in a log file.
  
  • Example: robocopy C:\source D:\destination /LOG:C:\log.txt

Complete Synchronisation Example with robocopy:

robocopy C:\source D:\destination /MIR /Z /COPY:DAT /R:3 /W:10 /LOG:C:\robocopy_log.txt

This command:

• Creates a mirror between the source and destination folder (/MIR),
• Enables the resume mode in case of interruption (/Z),
• Copies data, attributes and timestamps (/COPY:DAT),
• Limits copy attempts to 3 in case of error (/R:3),
• Waits 10 seconds between attempts (/W:10),
• Records the output in a log file (/LOG).

5. Remote Synchronisation on All Operating Systems with scp

Another common method for synchronising files on

Unix-like and Windows systems (via WSL) is the use of scp (Secure Copy Protocol), which uses SSH to securely transfer files.

Basic syntax of scp:

scp [options] <file> <user>@<host>:<destination>

Main options of scp:

• -r: Copy directories recursively.
  
  • Example: scp -r /source/ user@remote:/destination/
• -P port : Specifies a port other than the default port (22) for the SSH connection.
  
  • Example: scp -P 2222 /source/ user@remote:/destination/
• -i private_key: Uses a private key for authentication instead of a password.
  
  • Example: scp -i ~/.ssh/id_rsa /source/ user@remote:/destination/

6. Conclusion: Choosing the Right Tool for Every Need

Choosing the right tool depends on several factors, including the operating system in use, the need for local or remote synchronisation, and specific backup and security options. rsync is extremely powerful on Linux and macOS, while robocopy is the best ally on Windows for advanced copy operations. For remote synchronisation, scp is a secure and effective choice for all SSH-based systems.

Whether you are looking for a fast local solution or secure synchronisation between remote servers, these tools are the basis for efficient and reliable data management.