Sunday, June 9, 2013

Dockstar revive from death limbo


Dockstar

This post deals with reviving a Dockstar without any kernel booting(no usb reliable boot, no rescue system installed on flash, no stock pogoplug rom installed) but with at least access to Uboot by serial.
I will describe step by step how to do that and also configuring the whole system to get for future: netconsole uboot debug, install a debian rescuer on flash, make a backup of your whole system, get DHCP working,













Quick Steps:

I - Introduction
II - Description of the new debian USB stick
III - Upgrade Uboot to support kernel 3.2
  1.  Serial link to chat with uboot
  2.  Setup a TFT link to update uboot
  3.  Upgrade uboot
  4.  Configure uboot to work with dual partition(ext2, ext4) booting
VI - Reinstall Rescue system
VII - Add net console uboot ability
VIII - Configure network (dhcp to get ip, gateway, and dns auto configured)
IX - Conclusion


I - Introduction

Dockstars are normally sold as NAS with cloud abilities but regarding to its price (most of time >40€ (got mine for 15€)) its a very low consumption, and it hardware spec it make this device a perfect candidate to get an always online server at home (specs here).
The community and particularly Jeff doozan got very interested in hacking this device to get your own distro running on it, and they succeeded in hacking it tuto.
After about 3 years running 24/24 - 7/7 my dockstar died. RIP
I tried several times to flash the rescue system following this thread but without success.
The last solution, was to get a running distro from usb, and in a second time install rescue system to avoid problems I'm currently experiencing.


II - The new Debian - Usb Stick - 32 Go boot(ext2), rootfs(ext4):

Thanks to mister pluc (www.pluc.fr), I got an USB stick with a brand new kernel and a rootfs formatted into ext4(resiliant to hot power down...) with a kernel 3.2.0-3-kirkwood, and Debian 3.2.23-1.

The usb stick is partitionned and formatted as follow:
/dev/sda1        100M  ext2    boot
/dev/sda2        1G    swap
/dev/sda5        31G   ext4    rootfs

In /dev/sda1:
This partition is formatted in ext2 so as to be loadable by uboot. Uboot will look for a file /boot/vmlinuz in this partition flagged as boot. So inside this partition you will find the content of the /boot folder but like uboot is looking for the "vmlinuz" and "initrd" in this folder you need to make a symbolik link to self in this folder. Thanks to that uboot will look in this partiton and found:
/boot/vmlinuz
/boot/initrd

$ ln -s . boot
Once booted this partion is mounted on /boot

In /dev/sda2:
The swap partion to be used as ram if no more real RAM is available

In /dev/sda4:
The Rootfs partition formatted in ext4. This partion is mounted by fstab on '/' folder.
# /etc/fstab: static file system information.
#
# <file system> <mount point>  <type> <options>    <dump> <pass>
  tmpfs               /tmp                 tmpfs        defaults           0            0
  /dev/sda5           /                    ext4         noatime,defaults   0            1
  /dev/sda1           /boot                ext2         noatime,defaults   0            2
  /dev/sda2           none                 swap         sw                 0            0


III - Upgrade Uboot to support kernel 3.2

The remaining tricky part is so as to be able to boot on a kernel 3.2.X. I needed to reinstall a newer version of uboot (>=2011.12) from the current uboot (version 2010.09).
(FYI This re-installation need is due to a differences of L2 cache management between uboot exit state, and kernel waiting/needs which would result in a kernel panic due to corrupted data if the need is not satisfied.)

1 - Serial link to chat with uboot

To set up the conection between the PC and the dockstar before getting any terminal, we will need a USB to serial adapter, those device are recognised by the system as serial port and we will use minicom to send/receive uboot input/output
You can find USB serial adapter 5V on dealextreme (ex: http://dx.com/p/usb-to-uart-5-pin-cp2102-module-serial-converter-81872)

a - Host

Use minicom with /dev/ttyUSB0, change default device to use /dev/ttyUSB0:
minicom -s

Before going farer, verify the minicom's connection. You can shunt with a wire the TX and RX of your serial interface and type thing in your terminal, this should echo back your charaters... if not verify your connexion.
1 - Connect the USB serial
2 - Launch minicom
$ minicom

b - Dockstar

Connect the serie link to the dockstar following this schems:
dockstar | Serial USB
---------+-----------
GND      | GND
TX       | RX
RX       | TX



Dockstar and USB serial stick

Dockstar serial port

c - Talk to Uboot

1 - Un plug dockstar power supply
2 - Connect the USB and the wiring on the dockstar board header.
3 - Connect the USB
4 - Open a minicom connection
5 - Supply the dockstar
Once plugged on you should see appearing on minicom screen uboot launching message and autoboot count down press enter and access to uboot.


2 - Setup a TFT(Trivial File Transfert Protocol) link to update uboot

This protocole is designed to be the simplest as possible and the lighest, you even not find a command to list content of a folder. Thanks to this lightness it can be implemented in a bootloader that must be size efficient.
TFTP will be used so as to the bootloader(uboot) download from a remote host(my PC) the uboot update binary.

1 - On the Host

Get TFTP server and client:
$ sudo apt-get install tftpd-hpa tftp

Configure your server:
$ sudo vim /etc/default/tftpd-hp

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp" -> /home/sheda/work/dockstar/uboot/
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure"

You must have the uboot image in the root folder of the server, so change the folder or copy the image. (if you update the image you ne to restart tftpd deamon to reload config)
$ sudo service tftpd-hpa restart

Test the server:
Test the connection locally, by creating a temporay folder and download TFTP files from and to the PC.
$ cd /home/sheda/work/dockstar/uboot/
$ mkdir tmp_test
$ cd !$
$ tftp <your own host ip>
tftp> get uboot.mtd0.kwb-2011.12-dockstar-L2Coff
Received 529377 bytes in 0.1 seconds
tftp> q
$ md5sum uboot.mtd0.kwb-2011.12-dockstar-L2Coff
e18a2c7e308c249bc16f55bbaad31f50  uboot.mtd0.kwb-2011.12-dockstar-L2Coff

2- On the Dockstar Uboot

Before changing anything in the uboot environment you shout make a backup of it
u-boot>> printenv
=> Save the output by copy/past into a safe file.

Set the server host ip
u-boot>> setenv serverip <your host ip>

Set the dockstar ip
u-boot>> setenv ipaddr   <unused static ip for the dockstar>
<no need to save this into flash because this will not be used to auto boot>


3 - Upgrade Uboot by reinstalling it

This link is a very great tutorial with the good binary you want to flash on mtd0 from Jeff doozan which is dooing a very great work on hacking low power linux devices.
I will only explain the principle of the update because it already very well explained on jeff doozan forum.

The principle:
1. Get an image of uboot on your laptop/PC host
2. Get a tftplink up between your laptop containing the uboot image
3. Load into memory the uboot image
4. Erase the mtd0
5. Flash the new uboot from memory to mtd0
6. Reboot the dockstar


4 - Configuring uboot 2011.12 to boot on usb

Now uboot 2011.12 is fully working you need to configure it to load/launch correctly your brand new debian.


FYI, The boot sequence

Uboot will look sequentially on several support to boot:
- Usb
- Flash
- Stock pogo rom

USB will boot in severals steps:
1. Detect usb drives attached(usb start)
2. Test if one is bootable by searching for /boot/uImage on each partitions(run usb_init -> run usb_scan)
3. Set booting arguments (run usb_set_bootargs) with root(rootfilesystem), rootfstype, console
4. Load /boot/uImage in memory at 0x800000
5. Load the inird into memory at 0x1100000 if it is found
=>according to how your kernel is built you need or not inird utility, but in all case if kernel is built with initrd need, it will crash if it is not given at boot.
6. Really boot the kernel:
- with inird (bootm 0x800000 0x1100000)
- without inird (bootm 0x800000)


Settings

Usb_scan command scratching usb_root:
In the booting process the command that check usb device to find a booting one was scratching at boot the usb_root config, making the kernel not finding the rootfs:
usb_scan=.....setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done

1 - Make a backup:
u-boot>> setenv usb_scan_old 'usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done'
2 - And delete it:
u-boot>> setenv usb_scan 'usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done'

Give the kernel info on it root file system

u-boot>> setenv usb_root /dev/sda5
u-boot>> setenv usb_rootfstype ext4

[1rst time optionnal] Save evolution into flash:
You are not required to save now into flash your configs but you can test the boot and redo the same setting a second time with flash saving if the boot well completed.
So if you want to only do a test go directly to Reboot step, without saving evolution in flash.
Nevertheless, because the nature of the memory where configs are savedi(RAM) you have to reboot by software(reset cmd), reboot by unplugin/plugin power would result in losing config
u-boot>> saveenv

Reboot:
u-boot>> reset

Login Screen
Now you should be able to get the kernel booting, and obtain a login screen. If you choose to not save your configs in Uboot just restart the dockstar, re-follow instructions and save into flash this time.


VI - Install rescuer system on Flash

This "Rescuer" is a light debian system working from Flash, that allows you to restore a sytem, or re-hack an usb stick if yours burn or blew up. This is not mandatory to have one installed but it can save your time and money if problem happens on your usb stick.
This time it will be more simple that uboot update just follow jeff tuto and use its binary

Boot from an USB device (or a Debian NAND install) and run the following commands:
$ cd /tmp
$ wget http://jeff.doozan.com/debian/rescue/install_rescue.sh
$ chmod +x install_rescue.sh
$ ./install_rescue.sh
This script will download firmware to flash, test if you have an up to date version of uboot installed (we already did that :-) ), and it will do it in /tmp
If like me you configured a little to small(Fatality...) your /tmp you may no be able to download all required binaries, so create a temporary folder place you inside it and edit the script to change all /tmp/ by /<your temporary_folder_path>/ and launch the script.
(Source)

Test you system rescuer
1 - Power down your dockstar
2 - Un plug your USB stick
3 - Power up you dockstar
4 - Let it boot until the awesome blinking led alternating between green/orange for rescue system.
Now, You should be able to connect by ssh (log:root/pw:root):
ssh root@<dockstar_ip>


VII - Add Netconsole to the dockstar:

This is important so as to easily do maintenance on the dockstar if issues appears.
The principle is to inform uboot that you want input and output not on serial but on ethernet by shoouting every thing by UDP on port 6666 of a remote machine(the host, my PC)
This netconsole will take priority over the serial console if a ping to the host machine is ok.

So, if you experience some trouble with it just boot without the Ethernet plugged connected, serial will be used as console.
(Source)

Configuration:
To side are to be configured host side and dockstar side.
Dockstar can be configured by to ways, configuring from the Running Debian from the USB stick/NAND. Nor setting directly from uboot.
The PC has no particulare configs to do, we will use the nc tool.


Dockstar Side

From your running debian
$ fw_setenv serverip <host ip>
$ fw_setenv ipaddr <dock_star ip>
$ fw_setenv if_netconsole 'ping $serverip'
$ fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
$ fw_setenv preboot 'run if_netconsole start_netconsole'
fw_print/fw_setenv are only uboot-utilities useable from a running distribution, and do the same job as printenv/setenv launched directly on uboot shell.

From Uboot:
uboot setenv serverip <host ip>
uboot setenv ipaddr <dock_star ip>
uboot setenv if_netconsole 'ping $serverip'
uboot setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
uboot setenv preboot 'run if_netconsole start_netconsole' 
uboot saveenv


Host Side

nc -l -u -p 6666

Note: the above command will start a 'view-only' console. If you want to be able to interact with uBoot, you need to run the first 'nc' instance in the background and then use a second instance to send the commands back to the Dockstar:
nc -l -u -p 6666 &
nc -u 192.168.1.100 6666

Once finished, Run killall nc to kill the background nc process
killall nc

Now, when you switch on the dockstar:
On ttyS0 (Serial/JTAG dockstar's port) you will get very beginning of uboot process displaying that netconsol found the host on the network.
On the host with netconsole you will be able to get uboot prompt to dialog (you have to halt the boot process by pressing any key a the beginning when asked)
On the ttyS0 you will get the kernel boot output and a log in prompt



VIII - Get network running

This part is more a notes about having network working again. To understand what you need to debug your own problem is a sketch of how network is working on linux.

1 - Setup
Static:
Normally a network machine doesn't need dhcp to be able to communicate by setting statically its ip, gateway, dns it may be able to communicate
But we will prefere DHCP solution because it's more flexible

DHCP:
It's a service provided by most of router that allows a device to connect on the network and obtaining every thing the device need to communicate:
- IP => Its own adress on the network to send/receive packets
- GATEWAY => Normally the only device connected, Packets are send to it and it will route to the righ place( most of time outisde the LAN on the WAN)
- DNS => When your Browser whant to access http://sheda1805.blogspot.fr/ it need to resolve which ip is associated with this URL before connecting on it
To handle this protocol, an utils exist named dhcpcd. It may not be installed even if it is quite rare on nowaday distro.

2 - The principle
- Eth0 is binded every time on the same report to it's MAC address
- Eth0 will use dhcp protocol
- Allow right to dhcp to change network config for eth0(normally granted since installed)
- Reload network daemon (that sould reload configs, and get connection parameter from DHCP)

3 - Usefull set of usefull networking commands:
(source https://help.ubuntu.com/12.10/serverguide/network-configuration.html )
Clean the /var/lib/dhcpcd/* and reload networking
/etc/udev/rules.d/70-persistent-net.rules => Rule to assign logic name to networking peripheral according to its MAC
ifconfig -a | grep eth                    => Get all lan boards logic names
sudo lshw -class network                  => List all hardware for networking
sudo ethtool eth0                         => Get info on networking cards
/etc/init.d/networking reload             => Restart the network service by reloading the config, and resetting network peripheral
vim /etc/resolv.conf                      => Where to stock namesserver used to get ip from urls (Should be your gateway if you have a box)
vim /etc/default/dhcpcd                   => To configure dhcpcd daemon (allow overwrite the resolv.conf)
/etc/init.d/networking reload             => Test your changes
vim /etc/dhcpc/resolv.conf                => What is got by dhcp
/etc/network/interfaces                   => Tels the system how to configure the interface (eth0,1...)


IX - Conclusion

The dockstar is a great little server, and the work of Jeff doozan make it a pseudo reliable system, as a lot of utils, tools, tuto exist to debug problems that may appears.
Nevertheless I recommand severals doctstar best practices:
- Use the same file system architechture as mine and type due to the nature of the device hot shutdown may appear and the file system MUST be reliable to cop bad effects, EXT4 is a great candidate.
- Make time to time backups of the root usbstick(with dd tools) as it cost only time and a little HDD storage, but in the end if the system crash everything is not lost and it can be easily respawn by copying back the backup image onto the USB flash drive.
- Flash a rescue system would have been more than apreciable to reflash a bootloader and so I do recommand to take time to flash it onto the NAND memory

This took me a little bit of time to find a solution to my problem but thanks to that I discovered a lot of interesting tools (tftp, netconsole, uboot, linux boot process, linux kernel files, etc...).
Let me know if you have any questions,
Cheers,
Sheda

1 comment :

  1. Il est trop bien le site de http://pluc.fr :-)

    --
    Pluc, qui a toujours sur lui un adaptateur RS232-USB

    ReplyDelete

Let your mind talk