SD CARD / kernel issues?

[PlugPBX Admin]

So I just found my PlugPBX not responding to HTTP, or SSH (let alone calls).

It will answer pings.

Connected to it via serial-> USB (prior to rebooting) with another SheevaPlug I had and got the following repeating over and over.. (incrementing)

Code:

[3604963.770000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604963.770000] EXT2-fs error (device mmcblk0p2): read_block_bitmap: Cannot read block bitmap - block_group = 14, block_bitmap = 458752
[3604963.790000] Remounting filesystem read-only
[3604963.790000] mmcblk0: retrying using single block read
[3604964.810000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604964.820000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604964.830000] end_request: I/O error, dev mmcblk0, sector 2708988
[3604965.840000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604965.850000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604965.860000] end_request: I/O error, dev mmcblk0, sector 2708989
[3604966.870000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604966.880000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604966.890000] end_request: I/O error, dev mmcblk0, sector 2708990
[3604967.900000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604967.930000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604967.940000] end_request: I/O error, dev mmcblk0, sector 2708991
[3604968.950000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604968.960000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604968.970000] end_request: I/O error, dev mmcblk0, sector 2708992
[3604969.980000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604970.000000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604970.010000] end_request: I/O error, dev mmcblk0, sector 2708993
[3604971.030000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604971.040000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604971.050000] end_request: I/O error, dev mmcblk0, sector 2708994
[3604972.060000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604972.070000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604972.080000] end_request: I/O error, dev mmcblk0, sector 2708995
[3604973.090000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604973.100000] mmcblk0: retrying using single block read
[3604974.110000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604974.120000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604974.130000] end_request: I/O error, dev mmcblk0, sector 4119868
[3604975.140000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604975.150000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604975.160000] end_request: I/O error, dev mmcblk0, sector 4119869
[3604976.170000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604976.180000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604976.190000] end_request: I/O error, dev mmcblk0, sector 4119870
[3604977.200000] mvsdio mvsdio: FIFO_EMPTY bit missing
[3604977.210000] mmcblk0: error -110 sending read/write command, response 0x400d00, card status 0x400d00
[3604977.220000] end_request: I/O error, dev mmcblk0, sector 4119871
[3604978.230000] mvsdio mvsdio: FIFO_EMPTY bit missing

If anyone else gets this post here. I had an uptime of about 5 weeks, everything was running smooth prior.

Going to start looking at logs, and googling around. See if it comes back with a power off / restart, will check to see if SD card is firmly seated etc. (it was not moved or touched to my knowing)

Update - After cycling power it came back up, nothing in syslog or messages logs (of course if the SD kernel stuff dropped off it would not have been able to write any further logs of an issue. Might have to setup my PlugPBX to syslog to another host to catch whats going on.

Any see this please post... I wonder if this is a kernel issue with using the SD card?

[PlugPBX Admin]

Hmmm.

http://plugcomputer.org/plugforum/index.php?topic=1038.0
http://plugcomputer.org/plugforum/index.php?topic=897.0

I've seen the issue with not booting cold with some Class 10 media (advised user to use the double mmcinit trick in u-boot.

In this case, this PlugPBX is on a UPS so it didn't get a power reset issue, or I'd have had errors and other systems report this to me off the same UPS.

This is going to be something to watch. This particular system is using a 4 gig A-Data SD card (class 6). I've had no prior issues before.

I really wonder if there is something in the 2.6.30-2-kirkwood kernel at fault here, or if its a media issue with the SD card.

Still investigating. Anyone who see's this occur please post, and if anyone has more insight shout out please.

[PlugPBX Admin]

Looks like some homework ahead...and perhaps some enhancements to next release..

Taken from ...  http://plugcomputer.org/plugforum/index.php?topic=1074.0

When using SD cards under Linux, it seems to be a good idea to use ext2 vs. ext3 because ext3 causes even more writes for the journal. You should also consider using the deadline elevator for SD cards, telling it to focus on reads while collecting write requests in the cache for some time. I'm using the following script in /etc/rc.local to get this done when booting:

For the record, the SD card image provided is formatted ext2 for this exact reason above. I also have swap turned off on mine by placing the swapoff
command in rc.local as well (likely included in next release)

Code:

for i in /sys/block/mmcblk*/queue; do
  echo deadline >$i/scheduler
  echo 5000 >$i/iosched/write_expire
  echo 500 >$i/iosched/read_expire
  echo 1 >$i/iosched/fifo_batch
  echo 4 >$i/iosched/writes_starved
done

FYI - I've applied this to my rc.local and ran the script to see if it helps. Lets see if the PlugPBX can run more then 5 weeks issue free....

This is what mine looks like right now ...

Code:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Turn of Swap Disk - Don't need it with this systems footprint
# Setup delayed writes to the SD card

swapoff -a

for i in /sys/block/mmcblk*/queue; do
  echo deadline >$i/scheduler
  echo 5000 >$i/iosched/write_expire
  echo 500 >$i/iosched/read_expire
  echo 1 >$i/iosched/fifo_batch
  echo 4 >$i/iosched/writes_starved
done

exit 0

[PlugPBX Admin]

A clue?

Bad partitioning (doh) and system eventually tries to write to areas of disk not existing?

http://forums.plugpbx.org/index.php?topic=22.0

Anyone want to help with the  'math' and seeing if this is correct, also anyone know how to alter the partition to match the disk size (clean up the mess non-destructively?) - Need to find a clean in band solution for users hopefully (worse case build new release)

[alokprasad]

1.

When using SD cards under Linux, it seems to be a good idea to use ext2 vs. ext3 because ext3 causes even more writes for the journal. You should also consider using the deadline elevator for SD cards, telling it to focus on reads while collecting write requests in the cache for some time. I'm using the followingh script in /etc/rc.local to get this done when booting:

---Dont think elevator algorithm changes are needed once you already proceeded with flashybrid script.M i right?

For the record, the SD card image provided is formatted ext2 for this exact reason above. I also have swap turned off on mine by placing the swapoff command in rc.local as well (likely included in next release)

--Here too swap partition is already not used, so we cn remove swapoff from rc.local.

[PlugPBX Admin]

Correct on both points

[wkelleh2]

I think I've experienced this twice in the past few weeks.  I haven't connected to the serial terminal to verify your findings, but the symptoms were the same both times.  Did you fix this?

[PlugPBX Admin]

Yes, its fixed by using flashybrid. See the posting under the DEV forum.

I have yet to spin a final flashybrid edition of PlugPBX (waiting on a third sheevaplug to arrive to build and test - as my existing plugs are in use full time - wife gets pretty testy when I take the phones down to 'test stuff' )   

Basically follow the instructions listed here:

http://forums.plugpbx.org/index.php?topic=32.0

You have to apt-get install a script, configure some files, and tweak a few names of some startup scripts - its all well documented and many are using it now without issue (including myself)

Thanks for your patience!