cloud-and-crowd

 
about me     cv     research     blog    

Data sharing between multiple computers is as old as desktop computers themselves, for example look at first computers Steeve Jobs presents 1. That was really outstanding achiewment that you can collaborate and share changes, but there is a catch - you need to maintain your own personal server. Although that is easy and still effiecient for memebers of the same companny that is hard for your own use to collaborate with others. That is where Dropbox had made benefit our society with easy to use tool for society since 2007.

I have been Dropbox user for a long time and can recommend it for its stability and ease of use. But it cames with its tradeoffs - the space is limitted until you subscribe to their service. That felt too expesive for me and I started to look for alternatives in this April mainly what could I do with my raspberry pi and home ethernet connection.

Hosting my own data storage service like SAMBA or OwnCloud is hard. First of all I do not have an static ip address at which I could always connect which can be subscribed for additional fee from my ethernet provider. Also my netwoerk connection is not fast enough for uploading data (about 200 kb/s) so large data storage and managment seemed painfull. Thus costs for subscribing to Dropbox is actually a reasonable costs compared to your own effort for maintainig similar service.

A hope came when I read about Bittorent Sync technology. It allows to make syncronization folder which can be shared between computers (also phones) without use of cloud storage. That in turn offers limitless storeage and speed of sharing data. That is such an marvellous software to use! Thus although propertiary (Syncthing is promissing open source alternative) I decided to dive in it.

Setting up raspbian

Since data is not stored on cloud, for syncronizing at least two computers must be on. That is usually inconvinient thus as others I set my own allways on node on raspberry pi. Since every time when I setup a new stuff on raspberry pi I look for the same commands in the Internet I decided to make a detailed description for that here. I start with raspberry pi and SD card.

On the computer I download raspbian lite and make a burn to the SD card with a command

dd bs=4M if=2017-07-05-raspbian-jessie.img of=/dev/sdX

where X can be found from changes in "ls /dev". After that I create /boot/ssh file to enable listening for ssh conections for this one session (ssh file is deleted on boot). Then I push my SD card into Pi and attach it to my computer with ethernet cable with method "Shared to other computers" under "Network Connections". Ip adress can be found with

nmap -n -sP 10.42.0.255

where last number is Broadcast Address, which can be found under connection information. Another way is to scan whoole local network with a following command

nmap -sP 192.168.1.1/24

Now I am abl to connct to pi over ssh

ssh pi@10.42.0.240

where number is ip adress of previos step (password is raspberry). As a first step I change password with passwd command. Then I start raspi-config and do a following actions in the menu

  • increase partition size to occupy all SD card
  • Enable ssh on boot

Now raspberry is set up for configuration.

Finding ip adress by an email message

My internet provider proveides me with a public ip adress allowing to configure it outside my home network and other exiting things like personal VPN, Samba server, etc. However it is not static thus I have to keep track of it somehow. I have choosen to use email for doing that.

As ususal I spent long time to look for information for thing not possible (it was!!!) - sending email directly from ip adress to gmail. As a remedy I have to use an authorized (in a some way to avoid spam) email to also send message which requires to store my email password on the device. That is done by SMTP and the message script in python looks as follows:

#!/usr/bin/python
import subprocess
import smtplib
import socket
from email.mime.text import MIMEText
import datetime
# Change to your own account information
to = 'akels14@gmail.com'
gmail_user = 'akels14@gmail.com'
gmail_password = '*********'
smtpserver = smtplib.SMTP('smtp.gmail.com', 587)
smtpserver.ehlo()
smtpserver.starttls()
smtpserver.ehlo
smtpserver.login(gmail_user, gmail_password)
today = datetime.date.today()
# Very Linux Specific
arg='ip route list'
p=subprocess.Popen(arg,shell=True,stdout=subprocess.PIPE)
data = p.communicate()
split_data = data[0].split()
ipaddr = split_data[split_data.index('src')+1]
my_ip = 'Your ip is %s' %  ipaddr
msg = MIMEText(my_ip)
msg['Subject'] = 'IP For RaspberryPi on %s' % today.strftime('%b %d %Y')
msg['From'] = gmail_user
msg['To'] = to
smtpserver.sendmail(gmail_user, [to], msg.as_string())
smtpserver.quit()

To run this script on boot I put it in /home/pi/bin with "chmod +x" and add a following lines to "/etc/rc.local"

   _IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
  sleep 30
  sudo -u pi -i mailip
fi

That will execute the script after all other linux scripts are done. Sometimes I did not receive emails thus maybe one needs to set up rasdpberry for waiting for network until boot is finished which is available under raspi-config. Another option is to execute the script when network conection is established. That can be accomplished by putting the script in /etc/network/if-up.d/mailip. However that did not work for me.

Configuration of bittorrent sync

On the Internet I have read opinions that bittoretn sync 1.4 is the last stable version which works best with old raspberry pi which I did use. Also gui client on linux has not indicator applet on newer linux versions as development seems to be stalled 1. Fortunatlley after looking for an hour I found it on internet archive 2 (very great service!).

The arm archive downloaded in the link needs to be unarchived and palced and synced to raspberry pi which can be done with a command

rsync btsync pi@[ip pi]:~/bin/

It also needs to be started on boot which can be done by placing line before exit 0 in /etc/rc.local:

sudo -u pi -i nohup btsync --webui.listen 0.0.0.0:8888 &

To log in to the btsync in the browser a following address must be used:

[ip of pi]:8888/gui/

In there you can either craate or attach yourself to the sahres. And thats it you may now restart the device and it will still work as cloud for you.

Mirroring with cloud

After I used this setup over a month I found it to be very robust and also faster in synchronizing than Drobox is. The problem at the present is that almost noone uses if you compare userbase of Dropbox, Onedrive and Google drive. That in turn adds complexity where my pc may fail and makes it feel crowded. It felt that better option is to put the syncronization ovf various cloud services on my raspbery.

As it turned out that was geck of a pain. For example Dropbox client does not work on arm devices and there are no plans to supprot them. A far fetched option was to emulate it with exager desktop which worked in principle but was so slow. Then I turned to rclone a synchronization command line applicaction. ALthough it does its task great of making rsync like data trasnferrs it was no imediatly apparent how could I make syncronizartion of this one. It took me days (or weeks) until I found a way with "union-fuse".

Fistly I create three folders for example to dropbox:

mkdir -p ~/Cloud/Dropbox.diff
mkdir -p ~/Cloud/Dropbox.ro
mkdir -p ~/BtSync/Cloud/Dropbox

Then with unionfs I bind them together into /etc/rc.local (again before exit 0)

sudo -u pi -i nohup unionfs-fuse -o cow Cloud/Dropbox.diff=RW:Cloud/Dropbox.ro=RO BtSync/Cloud/Dropbox &

where union-fuse must be installed with "sudo apt-get install union-fuse". This command now will merge not intended to be written folder Dropbox.ro with differences Dropbox.diff to form an ordinary Dropbox folder which is synchronized over the network with btsync. What remains to be done is to push chnages from Dropbox.diff to cloud and synchronize Dropbox.ro with it. That is exactly I am doing.

To use rclone on pi I download binary at 3 for arm devices and put it in ~/bin on pi:

rsync rclone pi@[ip of pi]:~/bin/

Then I ssh on the pi, run "rclone config" and follow instructions to add remote like Dropbox. To test my remote I mirror cloud locally with a command

rclone sync Dropbox: ~/Cloud/Dropbox.ro

The last step is to put rclone and union-fuse together for forming a synchronization. The hardest part was to ensure that deletes remotelly or locally takes effect. Otherwise synchronization script is plainly simple:

#!/bin/bash
# The script which applies the changes from unionfs and pulls new data from cloud
Remote=$1 # Dropbox
BaseDir=$HOME/Cloud/$Remote
mkdir -p $BaseDir.ro && mkdir -p $BaseDir.diff

if test "$(ls -A $BaseDir.diff)"; then # A better option would be to use find to eclude sync files 
    mkdir -p $BaseDir.diff/.unionfs && cd $BaseDir.diff/.unionfs || exit 1

    find . -name "*_HIDDEN~" -type f | while read file; do rm "$file" && rm "$BaseDir.ro/${file:0:-8}" && rclone delete $Remote:"${file:0:-8}"; done
    find . -name "*_HIDDEN~" -type d | while read dir; do rmdir "$dir" && rmdir "$BaseDir.ro/${dir:0:-8}" && rclone rmdir $Remote:"${dir:0:-8}"; done 

    cd $BaseDir.diff || exit 1

    rsync -ua --exclude "*.!sync" --exclude ".unionfs" . "$BaseDir.ro"
    rclone move -u --delete-after --exclude "*.!sync" --exclude ".unionfs" . $Remote:

    find . -type d ! -name "*_HIDDEN~" -empty -delete
fi    

rclone sync $Remote: $BaseDir.ro
exit 0

which I place under ~/bin/rclone-bisync with usual chmod +x.

The last step is to make to execute this script regullarly which I do with crontab. That works as follows. I ssh into the raspberry and execute "crontab -e" which lets me to edit users crontab file. In that I put a following line

* * * * * sudo -u pi -i flock -n Dropbox.lock rclone-bisync Dropbox

which ensueres that synchroniation is being started every minute if it is not already running. But for this to be effective system must log in into user thus I set up autologin with raspi-config. Anothe r option is to edit system's crontab with "sudo crontab -e" which would not need autologin.

Improvements

References


This website was made with Skeleton with modificactions from JuliaDiff.