Howto: Raspberry Pi Root NFS share - boot your System over NFS share and definitively deal with Flash data corruption

*** Updated March 14, 2013  ***

The Goal: 

If you have a Raspberry Pi and a Linux Server (or a NAS), you shall really be interested by this post! :-)

I had a lot of issue with my main Rpi when overclocked generating File System data corruption...

And finally, the real best solution, solid and efficient has been to convert my root installation into booting rootfs over NFS.

I recommend the most easy solution to first have a running installation of your system into your Flash card and simply migrate it to root fs over NFS.

Since i've done this, never had any system freeze, corruption or event kernel panic with my Rpi overclocked to turbo mode :-)

Also, you should ensure before beginning that your system is up to date (sudo apt-get update && sudo apt-get dist-upgrade -f) and your have the last firmware version (sudo rpi-update)

Major source: 

• http://raspberrypi.org/phpBB3/viewtopic.php?f=63&t=5974
• http://raspberrypi.stackexchange.com/questions/628/how-do-i-configure-the-raspberry-pi-to-boot-with-an-nfs-root

Summary of steps: 

Step 1: Set your NFS share
Step 2: Copy your root fs into your NFS share
Step 3: Modify your Raspberry Pi boot configuration
Step 4: Adapt your Rpi fstab
Step 5: Boot your Rpi!

Step 6: Correct your swap configuration by migrating to a loop device

Memorandum

Step 1: Set your NFS share

If you have a Linux Home Server or NAS, then you probably already share data using NFS.

To set a NFS share dedicated for your Rpi Root fs, add your share into "/etc/exports":

Adapt <raspberrypi_ip> with your Rpi LAN IP or LAN subnet if you prefer

# Raspberry Root FS                                     
/data/rpi_rootfs <raspberrypi_ip>(rw,sync,no_root_squash,no_subtree_check)

Under Debian / Ubuntu, reload your NFS server config:

sudo /etc/init.d/nfs-kernel-server reload

Step 2: Copy your Rpi root fs into your NFS share

Then simply copy all of your Rpi root fs into your new nfs share, you can do it directly under the Rpi or by plugging your Flash card into a client computer:

Example with a client computer having the flash card and NFS share mounted:

cp -rav /media/mmcblk0p2/* /data/rpi_rootfs/ 

Step 3: Modify your Raspberry Pi boot configuration

The only partition you will need to keep in your Rpi Flash card will be the boot partition (first partition), containing main boot configuration files and the Rpi firmware.

To boot over NFS, we need to modify the file "/boot/cmdline.txt" (contained into the first fat partition of your Flash card) to add/correct some sections:

• root= --> Will be pointing to "/dev/nfs"
• nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3 ip=dhcp (replace<nfs_server_ip> with your NFS server IP)
• rootfstype=nfs
• smsc95xx.turbo_mode=N --> is a workaround to prevent kernel panic under high network load (i recommend this)

Note:

In this example, we use DHCP to set the Rpi Lan IP at boot time, this is in my opinion the easiest way to do as you preset a fix address in your DHCP server for your Rpi.

Still you can also manually a fix IP at boot time.

Also note we will be using NFS V3 running under UDP for better performances. (see Memorandum for performances fine tuning)

"cmdline.txt" example with DHCP (the file must contain only one line):

Replace:

• <nfs_server_ip> with the NFS server IP

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=dhcp rootfstype=nfs smsc95xx.turbo_mode=N 

"cmdline.txt" example with Fix IP(the file must contain only one line):

Replace:

• <raspberrypi_ip> with the Lan IP of your Rpi
• <nfs_server_ip> with the NFS server IP
• <default_gateway> with the IP of your local gateway

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=<raspberrypi_ip>:<nfs_server_ip>:<default_gateway>:<mask>:rpi:eth0:off rootfstype=nfs smsc95xx.turbo_mode=N

Step 4: Adapt your Rpi fstab

Now edit the Rpi "/etc/fstab" file before trying to boot, do under your NFS server or client computer.

/data/rpi_rootfs/etc/fstab:

• Delete the original line corresponding to your root fs and pointing to the second partition of your flash card  (/dev/mmcblk0p2), we don't need anymore as it will automatically be mounted by the firmware at boot time

Step 5: Boot your Rpi!

Ok, let's go, time to boot :-)

If you follow all steps carefully, you system should boot with no major issue.

Therefore, you will not have anymore swap available, by default Raspbian uses dphys-swapfile os use a local file as swap.

We will correct this now.

Step 6: Correct your swap configuration by migrating to a loop device

By default, Raspbian uses dphys-swapfile to generate a local file being used as swap, this won't work anymore when booting under NFS.

I don't recommend to use your Flash card as a swap partition, this may generates system freeze or kernel panics if you have data corruption:

The better way is to set a local file as a loop device that will be used a swap device, here is how.

Clean current non working swap file and uninstall dphys-swapfile:

sudo apt-get remove --purge dphys-swapfile
sudo rm /var/swap
sudo rm /etc/init.d/dphys-swapfile
sudo update-rc.d dphys-swapfile remove

Create a new swap file, create the loop swap device and activate swap (exemple with 1GB swap):

sudo dd if=/dev/zero of=/var/swapfile bs=1M count=1024
sudo losetup /dev/loop0 /var/swapfile
sudo mkswap /dev/loop0
sudo swapon /dev/loop0

Check your current swap availability:

$ free

Output example:

             total       used       free     shared    buffers     cached
Mem:        237656     213092      24564          0         24      93192
-/+ buffers/cache:     119876     117780
Swap:      1048572       1556    1047016

Make it permanent, edit "/etc/rc.local" and add this section before "exit 0":

echo "Setting up loopy/var/swapfile.."
sleep 2
losetup /dev/loop0 /var/swapfile
mkswap /dev/loop0
swapon /dev/loop0

Step 7: Other tunings

There is also some other little thing to tune:

Edit "/etc/default/rcS" and:

• add:ASYNCMOUNTNFS=no
  
  

Edit "/etc/sysctl.conf" and:

• add or set: vm.min_free_kbytes = 12288 

This will ensure the system will always have 12Mb or RAM free to prevent kernel panic risk, your may try lower value if your prefer.

Memorandum

• NFS version and fine tuning

You may want to try different settings to get the better performance possible.

First, if you test your write speed, using dd will be very easy:

Create a 10 Mb file test:

$ dd if=/dev/zero of=/tmp/test.file bs=1M count=10

Output sample:

10+0 enregistrements lus                                                                                                      
10+0 enregistrements écrits                                                                                                   
10485760 octets (10 MB) copiés, 1,52525 s, 6,9 MB/s

You may want to try with different NFS version, change in cmdline.txt:

• Change the section nfsroot "vers=2/3"

You may want to try TCP versus UDP

• Change the section nfsroot  "udp" or "tcp"

You may want to try different values of "rsize" and "wsize", example with NFS V3 and TCP:

Example:

• root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfsrsize=32768,wsize=32768,tcp,vers=3

In my case, it did not really change anything, so i kept kernel default values for wsize and rsize, udp with NFS V3.

• Netfilter Iptables

When modifying your Iptables configuration, keep in mind that NFS traffic with your NFS server will result in system halt.

Applying default outbound policy to DROP (usually "iptables -P OUTPUT DROP") will in system crash.

You should apply instead "iptables -P OUTPUT ACCEPT" which will permit any outbound traffic from your Rpi (not a big deal, usually you trust your own machine)

Also you can ensure to accept NFS traffic with your NFS server before applying any other rules.