Read/Write Hybrid Solution - Testers / Help Needed!

[PlugPBX Admin]

So as I posted here...

http://www.plugpbx.org/?p=367

I’ve got a special idea

Boot the system from SD card read-only – map to a RAM disk what needs read/write access (like the SQL database, and logs, and /etc).

Make some changes? No problem, just click a ’snapshot’ button on the PlugPBX web interface and the system remounts read/write and syncs whats changed back to your SD card – when done, mount it read only again.

Bullet proof. No SD card wear outs, lock your setup down – power interruption survivability. Use a cron script to sync changes back to SD card as you wish perhaps?

Need to apply an update? Flip the SD card back to read/write – do your apt-get fun and then throw it back to read-only with a single command.

Its called flashybrid (set of scripts). Some issues, but found some clever work arounds. I’m playing with this on my own phone system (eat the dog food) and if I can make this work safe, Its where I’m taking things – there is far too much risk with SD card media being used heavily, and this makes for a rock solid system.

SO.. Patience as I tinker away. I will release the same system with this add on for now, along with a roll up of fixes and bugs found to date when time and tests permit. Backup your setups and be sure to use quality media for now.

I am looking for some testers of this idea. The risk is that you could blow up. SO you'd backup your SD card to a spare before doing this. I'm trying this out tonight at home myself in about 10 minutes after this worked on a test bed)

1) apt-get install flashybrid
2) add "ENABLED=yes" to /etc/default/flashybrid
3) mkdir /ram (root of system)
4) Configure the three following files as follows:

/etc/flashybrid/config

Code:

# This is the main configuration file for the flashybrid system.

# The mount point of your flash memory.
FLASHMOUNT=/

# You probably want a ramdisk to be set up with directories that the system
# needs to write data to frequently, so that your flash disk can be mounted
# read-only. Flash memory can only be written to several thousand times over
# its lifetime, so setting up a ramdisk will extend the lifetime of your
# flash a lot. This is the location to mount the ramdisk. Comment out the
# line to not set up a ram disk.
RAMMOUNT=/ram

# Commands to run to enter full mode. The disk will be mounted before
# this.
FULL_CMDS="mount -o remount,rw /"

# Commands to run to enter embedded mode. The disk will be unmounted before
# this. If you wanted to spin a disk down, you could add a hdparm -y
# command.
EMBED_CMDS="mount -o remount,ro /"

# This controls the maximum ammount of memory you want to allocate  
# to the tmpfs RAM drive
# This parameter is optional, and if you do not supply if, tmpfs will
# occupy up to 50% of your available memory (ram+swap)
FLASH_MAX=192m

# You may also want to edit the other files in this directory:
# ramtmp, ramstore

VERBOSE=yes

/etc/flashybrid/

Code:

# This is a list of directories that are used to store variable data
# that is written to during normal operation of the system, and that should
# be preserved across reboots.
#
# On boot, the listed directories on your flash disk will be copied to the
# ram disk, and then bind mounts will be used to make the directories on
# the ram disk replace them.
#
# On shutdown, the contents of those directories on the ramdisk will be
# rsynced back to the flash disk.
#
# Directories listed here that are not present will be silently ignored.
# Do not put the trailing slash on directories!

# Should always be in ram disk.
/etc
/var/cache/samba
/var/lib/samba
/var/backups
/var/lib/asterisk
/var/lib/postfix
/var/lib/dbus
/var/lib/logrotate
/var/lib/mysql
/var/webmin
/var/log
/var/run
/root

# If you don't use devfs/udev, you will want /dev in ram, since many things
# need to modify it when the system is running. Not needed on systems
# running udev.
#/dev

# Necessary if the system is to be able to send mail while in embedded
# mode.
/var/spool
/var/mail

/etc/flashybrid/ramtmp

Code:

# This is a list of directories that should be set up as temporary
# directories in the ramdisk. On boot, the directories listed will be
# created on the ramdisk, mode 1777. Their contents will not be saved
# across reboots. The equivilent directories on your flash will be
# removed and replaced with symlinks to these directories.

# Do not put the trailing slash on directories!

/tmp
/var/lock
/var/lib/dhcp3
/var/lib/php5
/var/lib/misc
/var/lib/ntp
/var/lib/urandom

# This could go in ramstore if you prefer for its contents to persist
# across reboots.
/var/tmp

And then finally this dirty hack for now in /etc/rc.local

Code:

#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

#We have to restart syslog and cron to force them to start using our
#ramdisk binded mount points with flashybrid - it does not create them soon enough during init startup

/etc/init.d/rsyslog restart
/etc/init.d/cron restart
/etc/init.d/nfs-common restart
/etc/init.d/portmap restart

#We then set the system to run the root filesystem read only

mountro

exit 0

Then, take a deep breath, and reboot. The mount points to ram don't happen soon enough for the four services restarted in rc.local, hence restarting them which then has them open files in the ram disk. The "mountro" and its friend the "mountrw" are flashybrid scripts that set the root filesystem read only or read/write (so you can make changes dummy!)

Should bootup readonly, just type "mount" in ssh once fully booted up and see the following first line. Yah its gonna be messy, lots of stuff is binded to the ram disk mounts

Code:

/dev/mmcblk0p2 on / type ext2 (ro,errors=remount-ro)

SO in theory when you shutdown, changes are rsynced from ramdisk back to your SD card (not sure how reliable flashybrids stuff is) and you can trigger a sync with the 'fh-sync' script command anytime.

Looking for some testers, I think this or something close is the way to go. Makes unreliable SD card lifetimes fixed, gets us closer to a linksys router type of embeded setup, but you can still work with it like a normal server at a flick of a switch.

Idea is to perhaps sudo fh-sync and put a 'save now' button on the PlugPBX web page.


If anyone wanted to help me out, this is the big one I need feedback on from testers. Roll up and dive in.

[PlugPBX Admin]

A quick note. My estimates show around 70 meg free in the ram disk, you might have to make it larger then the 192meg I pulled off the top of my head but that gets riskier. I have not 'optimized' what I'm throwing into the ram disk (need to lsof and see whats not u or w type and prune perhaps).

I also know that asterisk is NOT rotating logs so you might want to setup webmin to do so for you (just google how to rotate asterisk logs) and say only retain the past weeks etc. My intent would be to preconfig all this on the next big release etc)

That's about it!

[PlugPBX Admin]

Crap.

Code:

PlugPBX:~# lsof |grep "179,2" |grep "w "
dhclient3  604       root    3w      REG      179,2     1078     159231 /var/lib/dhcp3/dhclient.eth0.leases

(FYI dirty way to see whats being written to SD card (use a u or w or perhaps a regex etc hehe)

So Mine won't go to read only mode at bootup. Somehow the dhcp3 guy did his silly thing before the ram disk bind setup. Ahh timing windows, scripts, boot orders. Fun stuff.

I might just throw more of this into fstab so it gets mapped/mounted at bootup without flashybrids scripting (just write my own rsync  scripts maybe?)

Hence testing with lots of volunteers and fellow geeks   I DO like the idea that people could easily 'add' their own stuff to the PlugPBX and just tell flashybrd to take care of it though..

Okay for now I just issued a 'kill' for the PID of dhclient and re-ran it,  (the lease info in /ram now) and i was able to issue the mountro command, and the root filesystem is now read only. So my own PlugPBX is now running read-only off a very crappy and failing SD card I have (dies within a week so far and needs power cycle) so we'll see how this handles that heheheh.

I'll see how this pans out I guess and look at making this 'dirty' into 'clean'. Time test if you can please!

[drvcycle]

My SheevaPlug is supposed to be delivered on Tuesday, so I'll see if I can start with testing this. I assume I can do the normal image flash my SD card and follow the instructions in this post to do the RAM disk stuff?

Dennis

[PlugPBX Admin]

Correct.

[drvcycle]

I finally received my Plug and flashed my SD card. Then I followed your instructions here. The system would boot up, but the FreePBX interface would not work - it was complaining about MySQL not working, or something. Then I remembered reading your posts on plugcomputer.org (I think) where it was mentioned that flashybrid needed to be made to startup before all the other services. So I renamed S20flashybrid to S01flashybrid in rc2.d, rc3.d, rc4.d and rc5.d. I also renamed K02flashybrid to K99flashybrid in rc0.d, rc1.d and rc6.d since I figured it would be good if flashybrid shutdown after all the other services had been stopped.

Then, after rebooting, I did as you suggested and killed the dhclient3 and restarted. This cleared up all of the open files on the root file system and I could successfully run the mountro command to get the root file system in read only mode.

I have not tried to configure any thing on the system yet, but I'll play with it some more tomorrow evening if I have time. Is there anything in particular that you would like me to test on my PlugPBX?

Dennis

[PlugPBX Admin]

First off, welcome to the party

Yes, I think placing the flashybrid script to run in the rc.s script prior to any run level scripts running might do the trick and beat out the dhcp service from preventing mounting root read only... Take stab at it

I have yet to try this myself but its on this weeks to do list.

[drvcycle]

I can check this out tonight. I'll remove flashybrid from the rc.x directories and add it to rc.S. I suppose I should leave flashybrid kill scripts in rc.0, 1 and 6 to ensure that it is shutdown properly when the system is shutdown? I'll let you know what I find out.

Dennis

[PlugPBX Admin]

You are correct sir - that's the idea.

If this works, then I can make it a standard part of PlugPBX.

Sudo the fh-sync script and link to a 'snapshot' button on the PlugPBX web interface, and users can 'save' their settings back to SD card, otherwise the system runs in ram disk only with a read-only SD media - bullet proof.

If some users want logs saved, they can syslog them to another host, and/or use a cron script to sync things back to SD card at an interval they are comfortable with.

PlugPBX will then live forever on SD media, including the cheap crap I'm using as a worse-case test bed currently.

[PlugPBX Admin]

WOOOOOOOOT!

Nailed it

Code:

PlugPBX:/etc/rcS.d# ls
README     S05hwclockfirst.sh    S07udev-mtab     S11mountoverflowtmp  S15mountnfs.sh
S01mountkernfs.sh    S06checkroot.sh   S08checkfs.sh     S12networking  S16mountnfs-bootclean.sh
S02udev     S07hwclock.sh   S09ifupdown     S12procps  S17bootmisc.sh
S03mountdevsubfs.sh  S07ifupdown-clean   S09mountall.sh     S12x11-common  S17urandom
S04bootlogd     S07module-init-tools  S10mountall-bootclean.sh  S13portmap  S18stop-bootlogd-single
S05hostname.sh     S07mtab.sh   S11flashybrid     S14nfs-common

ONLY start-up flashybrid before the network interface kicks in, remove it from any of the rc#.d folders, and stop it near the end of shutdown.

I took a guess, took a deep breath, rebooted the phone system and she came up, read only (with the rest in the ram disk). Its magic

SOOOOOO. Time to roll this into an official release along with a 'Save' Button to snapshot changes back to SD card.

PlugPBX just got rock solid SD card wise. Woot.

I putup a quick posting on PlugComputing.org for their benefit as well, as I think this should be a standard way to build any PlugComputer setup when you use SD cards as the root filesystem...

http://plugcomputer.org/plugforum/index.php?topic=1074.msg8720#msg8720

Do test it out boys in the meantime.

I'm having a Beer, this calls for brewski - Been musing over this problem from over a month now... hot damn its solved!    

[PlugPBX Admin]

Update:

I've applied this to some other Plug appliances I have.

Warning: Changing script ordering by hand, is broken once another apt-get install (of a service) occurs as they use the standard debian scripts to resolve dependencies and ordering. So I've seen it move the S11 to S20 ordering for the entry of flashybrid in /etc/rcS.d/ (and other scripts I had placed in by hand on another project)

SO, looks like I might have to read up on the bloody dependency scripting stuff used now for setting up all the rc stuff....otherwises users would break stuff by installing applications or updates potentially.

The good news is that the other system I have also easily is now running read-only at boot-up, so the big hurdle is over, now its just small details....

[drvcycle]

Cool! I added S11flashybrid to rcS.d as you suggested. I also changed my kill syscripts in rc0,1 and 6 to K09flashybrid so flashybrid would be stopped before the filesystems get umounted and the system reboots or halts. This will also make the root file system writable when in single user mode.

I rebooted my plug system and my test files were still in place when the system came back up, so it appears that changes are being flushed to the SD card before the system goes down. I'll configure some stuff in FreePBX and reboot and verify that those changes get properly flushed to SD before I make too many changes!

I'm not really familiar with the Debian package dependency stuff, but if I can find some time I'll see if I can learn anything that would be helpful to prevent flashybrid's start scripts from being messed with by apt. Do you know who is responsible for the flashbrid package? It seems like the flashybrid package needs to be updated since the way it get set up by default doesn't really make sense.

Thanks for your all your hard work!

Dennis

[PlugPBX Admin]

Yes, anyone who can point out / find / play with how to get Flashybrid (or debian rc.x scripts in general) how to play nice (how to set and lock down their priority) , that help would be fantastic, as its holding me back from making this official for everyone to use.

[regloh]

Greg,

a correct LSB Header in the script should solve the problem.

"X-Start-Before:" works since insserv package version 1.09.0-8 (IMHO plugpbx comes with 1.12.0-14) even this is not a standard-defined header,
its only a proposed extention.

More details can be found in the debian wiki
http://wiki.debian.org/LSBInitScripts

Holger

[invictive]

Ok, so I read the whole thread but I'm still a little confused...

I can set this up by following the instructions in the first post, but then what do I have to do to make it work?

Is this ready to go mainstream yet? I'm hoping to get this PBX up and running within a few weeks, so I'm itching to tinker.