IPG Doc

User Tools

Site Tools

Trace:
backup:laptops

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
backup:laptops [2021/03/10 11:43] – adminbackup:laptops [2023/10/09 12:33] (current) – admin
Line 4: Line 4:
  
 ** All Employee of EPFL are required to do regular backups of the laptop computer they use **\\ ** All Employee of EPFL are required to do regular backups of the laptop computer they use **\\
-For this reason EPFL provide a backup service (Druva InSync) free of charge that every user can request to use. +For this reason EPFL provide a backup service (aTempo Lina) free of charge that every user can request to use. 
-to get your Druva InSync account, please contact your system administrator, then, once installed and configured, the program will backup your files every 4 hours even if you're not connected to EPFL networks.+
  
 ===== Personal Laptops ===== ===== Personal Laptops =====
  
-The lab doesn't provide direct support to personal laptops, but you can use different services of our filer structure to your gain. The use of these services as backup system is under your only responsibility, we don't control what you do and how you do it. +The Unit IT staff doesn't provide direct support to personal laptops, but you can use different services of our filer structure to your gain. The use of these services, as backup system, is under your only responsibility, the unit do not control what you do and how you do it.
- +
-  - Homedir: the home directory is to store your work related documents/programs/stuff, but you can use it also to store some personal related data (photos/images, videos, documents) if the footprint is relatively small (under 100 GB). +
-  - Scratch server: The scratch server provide space for everyone and there's no limit on the use of it. please use it as you like, but adopt a fair play policy, as everyone in the lab will need to store data. +
-  - google disk/dropbox/microsoft onedrive: these and others internet services are offering storage space for your personal data, **but their storage/use policy is against the rules of EPFL, the European GDPR and/or the Swiss laws regarding privacy** . Please refer to official EPFL documentation/policies about the  use of cloud services. +
- +
-===== Syncronizing work stuff with Unison ===== +
-[[http://www.cis.upenn.edu/~bcpierce/unison/|Unison]] is a very fast, efficient and easy to use file-synchronization tool for Unix (Linux and Apple OsX) and Windows.  +
- +
-Note that, although very convenient, **Unison is not a real backup tool**: it does not keep an history of files, and it tends to replicate mistakes instead of reparing them (a file deleted by mistake on one machine will be deleted also on the other machine). On the other hand, we keep various backup of your home directory on the server. Therefore keeping your laptop in sync with your directory on our file-server is almost like backing it up. +
-  +
-Here we show how it can be used to keep the work stuff on your laptop in sync with your home directory on the file server. +
-Here is a short check list of the things to do: +
-  - install unison +
-  - cleanup your directories +
-  - chose and/or prioritize what to synchronize  +
-  - decide what to exclude from synchronization +
-  - setup unison configuration files and startup scripts +
- +
-==== Install Unison ==== +
-You can either download it directly from the [[http://www.cis.upenn.edu/~bcpierce/unison/download.html|official web site]], or use a pre-canned package from your favourite distribution. Unison is included in most Linux distribution and on both [[http://fink.sf.net|fink]] and [[http://macports.org|mac ports]] for Apple OS X. For a nice feedback from the backup scripts, we also suggest to install the [[http://growl.info/|growl notifier]] on OS X and libnotify on Linux. +
- +
- +
-==== Cleanup your files ==== +
-Let's classify our files as //static// or //dynamic// files, and as //personal// or //professional//. Static files are created once and modified at most very rarely as, e.g. pictures, mp3 files, or pdf of downloaded papers. Dynamic files are those that you edit often as, e.g. the latex source files of your latest revolutionary scientific paper, your program sources as well as your contact list. Off-course dynamic files are those that need particular attention and a more frequent backup. Luckly, dynamic files are typically much smaller than static ones. In fact, very often most of the space on the disk is taken by personal static data (pictures, movies and music collection). +
- +
-<note> +
-Please **do not use our server for storing your personal static data**. Buy yourself an external Firewire or USB hard-disk instead. The backup of our home directories on tapes takes a long time and the tapes have limited capacity. It is impossible to backup tens of GigaBytes of non-work-related data for every user. The largest the stuff to backup the less frequent and safe the backups are. Please, respect the other users by keeping our disk usage on the servers as limited as possible. +
-</note> +
- +
-==== Synchronization policies ==== +
-We suggest to use two different policies (two separate configuration files) for synchronizing your data.  +
-  - a frequent (hourly) and automatic synchronization for all your dynamic data (we accept here also your personal files if they are not too large); +
-  - an on-demand or less frequent for professional static files. +
- +
-==== What to exclude? ==== +
-Clearly any **file that is derived from another one** (e.g. compiled files like object .o files, and latex output .dvi files) should be excluded from any backup. Strictly speaking, synchronization is not a backup but excluding non source files will speedup your synchronization process. +
- +
-Version control systems can be seen also as synchronization systems. Therefore I (Giovanni) personally exclude also all the files that are under centralized version control (SVN) by adding the ''*_svn'' pattern to my exlude list and appending ''_svn'' to the name of any directory containing svn work directories. Note that this is a **potentially dangerous** thing (see below) and I will probably stop this practice as soon as I've migrated all my svn repositories to git. +
- +
-==== Configuration example ==== +
-Unison is easy to use, once the configuration file is set.  +
- +
-$ unison -batch myPolicy +
- +
-where ''myPolicy'' is the name of the policy to use and is described in a file with the same name under the ''$HOME/.unison'' directory. +
- +
-Here is how my main configuration file looks like: +
- +
-<code> +
-# Roots of the synchronization +
-    root = /Users/cangiani +
-    root = ssh://cangiani@algosrv5.epfl.ch/ +
- +
-# Here is what I want to keep in sync +
-    path = Archives +
-    path = Documents +
-    path = Learn +
-    path = Notes +
-    path = Projects +
- +
-    ignore = Path Documents/Local +
- +
-# Some regexps specifying names and paths to ignore +
-    ignore = Name temp.* +
-    ignore = Name *~ +
-    ignore = Name .*~ +
-    ignore = Name .o~$ +
-    ignore = Name *.{o,x} +
-    ignore = Name *.{tmp,aux,log,dvi} +
-    ignore = Name *.sparseimage +
-    ignore = Name .DS_Store +
- +
-# always use rsync for sending files +
-    rsync = true +
- +
-# first treat smaller files +
-    sortbysize = true +
- +
-# On Mac the default FS is case insensitive   :(    +
-#    ignorecase = true +
- +
-# Log actions to the terminal +
-    log = true +
- +
-# Specific settings +
-    key = 1 +
-    label = Learn directory +
-    batch = false +
-</code> +
- +
-Please refer to the [[http://www.cis.upenn.edu/~bcpierce/unison/download/releases/beta/unison-manual.html|official documentation]] for all the details. +
- +
-<note warning> +
-**Be careful with ignore files!!!** +
- +
-  +
-Here is an extract from the official documentation: +
- +
-It is important to be a little careful when renaming directories containing “ignore”d files.  +
-For example, suppose Unison is synchronizing directory A between the two machines called the “local” and the “remote” machine; suppose directory A contains a subdirectory D; and suppose D on the local machine contains a file or subdirectory P that matches an ignore directive in the profile used to synchronize. Thus path A/D/P exists on the local machine but not on the remote machine. +
- +
-If D is renamed to D' on the remote machine, and this change is propagated to the local machine, all such files or subdirectories P will be deleted. This is because Unison sees the rename as a delete and a separate create: **it deletes the old directory (including the ignored files) and creates a new one** (notincluding the ignored files, since they are completely invisible to it). +
-</note> +
-==== Do it! ==== +
-Now that you've prepared and tested your perfect configuration file, it is time to make sure that unison is executed periodically. On unix (linux and mac) you can symply call unison from a script like the following: +
-<code> +
-#!/bin/sh +
- +
-# make sure to use always the same hostname even if the actual address of the laptop is variable +
-export UNISONLOCALHOSTNAME=giovanniMBP +
- +
-# make sure to be at EPFL connected on a wired connection +
-ifconfig | grep 128.178.70 > /dev/null 2>&1 +
-if [ $? eq 0 ] ; then +
-  unison -batch main +
-fi +
-</code> +
-and launch the script periodically as a cron job. Edit your cron table (''crontab -e'') and add something like +
-<code> +
-38 8-20/2 * * * /Users/cangiani/Desktop/unison.command 2>&1 > /dev/null +
-</code> +
-which runs the script every 2 hours between 8:38 am and 8:38 pm. +
- +
-Note that the line  +
-<code> +
-export UNISONLOCALHOSTNAME=giovanniMBP +
-</code> +
-of the above script is quite important because the database used by unison for keeping track of which files are changed since last sync depends on the host-name (address) of your machine. Since the laptop gets a different ip address and, therefore, a different host-name each time it is connected to the network it happens often that you have false conflicts: a file is supposed to be changed on both machines only because it was synced on a previous run with a previous host-name. +
- +
-----+
  
-Here is a nicer version of the launch script that keeps you notified about the result of your automatic synchronization: 
-<code> 
-#!/bin/sh 
  
-export UNISONLOCALHOSTNAME=giovanniMBP+===== Access file servers using you Personal Laptops =====
  
-ifconfig | grep 128.178.70 > /dev/null 2>&1 +  - Homedir: the home directory is used to store your work related documents/programs/stuff, but you can use it also to store some personal related data (photos/images, videos, documents) if the footprint is relatively small (under 100 GB). 
-if [ $? -eq 0 ] ; then +  - Scratch server: The scratch server provide space for everyone and there's no imposed limit (time or space) on its use. Please use it as you like, but adopt a fair play policy, as everyone in the Unit will need to store data. 
-  ofile=$(mktemp /tmp/unison.XXXX) +  - google disk/dropbox/microsoft onedrive: these and others internet services are offering storage space for your personal data, **but their storage/use policy is not always in line with the rules of EPFL, the European GDPR and/or the Swiss laws regarding privacy** . Please refer to official EPFL documentation/policies about the  use of cloud services.
-  unison -batch main 2>$ofile 1>&2 +
-  if [ $? -eq 0 ] ; then +
-    msg=$(grep "Synchronization complete" $ofile; grep "Nothing to do:" $ofile) +
-#    DISPLAY=:0.0 notify-send -u low "Unison Sync OK" "$msg"                  # linux version +
-    growlnotify --title "Unison Sync OK" -p Low -m "$msg"       # mac version +
-  else +
-    msg1=$(grep "Synchronization complete" $ofile) +
-    msg2=$(grep '<-?->' $ofile | grep "new file" | head -n 1) +
-    msg3=$(grep '<-?->' $ofile | grep "changed" | head -n 1) +
-#    DISPLAY=:0.0 notify-send -u critical -t 0 "Unison Sync Err" "$msg1 $msg2 $msg3"        # linux version +
-    growlnotify --title "Unison Sync Err" -s -p High -m "$msg1 $msg2 $msg3"   # mac version +
-  fi +
-  rm $ofile +
-fi +
-</code>+
  
-{{backup:unison.sh|Here}} is the file I actually use.  
  
backup/laptops.1615376627.txt.gz · Last modified: 2021/03/10 11:43 by admin