Generate the list by logging into host A:

# dpkg --get-selections \* > apps.lst

Copy apps.list to host B and execute the following:

# dpkg --set-selections < apps.lst
# apt-get -u dselect-upgrade

If there are apps which need to be installed you will see the familiar apt-get/aptitude interface displaying the apps to be installed and a prompt to confirm. Once confirmed, the apps will be installed and both machines should be mirrors (from an application standpoint) of each other.

This article presumes one is running Linux (in my case Ubuntu 10.04 though in principal this should work for pretty much any Linux distro running a 2.6 kernel). I will describe the process for formatting the USB flash drive, mounting an ISO file to the local file system in order to copy files to the USB drive, and installing a bootloader and making the USB drive bootable.

While searching the web for similar solutions, I came across a few other blogs^[1][2] that describe the procedure, only to discover that they lacked the critical component for how to make the drive bootable. Therefore, I’m going to make my own attempt to cover this topic in a more structured, comprehensive fashion.

Prerequisites

There are a few software requirements that must be met before proceeding. Firstly, be sure to have a copy of the latest version of the g4l ISO file (you can download version 0.36 here.

Secondly, it will be necessary to have a bootloader available. In this article, I will be using syslinux. syslinux is a lightweight bootloader for MS-DOS FAT filesystems and will be used to generate the bootloader image. You can download the latest version here. I understand that GRUB or possibly LILO could be used, but for this article I will stick to syslinux

Lastly, be sure to have parted installed. parted is a command line tool for manipulating partition tables. This is useful for creating space for new operating systems, re-organizing disk usage, copying data on hard disks and disk imaging. This may already be installed with your distro, but if not, be sure to install it either from the package repository or from source.

Once the required software has been downloaded and/or installed, begin by opening a terminal window and mounting the g4l ISO to a local mount point. In the following example the mount point is the directory g4l created in the users home directory. NOTE: unless your system is configured to permit your user account to mount filesystems it will be necessary to have root privilege to mount the ISO. In the case of Ubuntu I use sudo, however it may be necessary to su root depending on your distro (your mileage may vary!)

$ sudo mount -o loop ~/downloads/iso/utilities/g4l-v0.36.iso ~/g4l
Next, expand the syslinux tar file. In the following example I expand the file into my home directory:

$ tar xjvf ~/downloads/iso/utilities/syslinux-4.03.tar.bz2
syslinux-4.03/
syslinux-4.03/version.gen
syslinux-4.03/MCONFIG.build
. . .
syslinux-4.03/linux/syslinux
syslinux-4.03/linux/Makefile
syslinux-4.03/version.mk


We are now ready to proceed with the real work!

Create a FAT partition on target USB drive

To being, mount the USB drive and observe the output from dmesg to determine what device identifier is assigned. Depending on your distro and if the drive has already been formatted it might auto-mount. If so, be sure to unmount the device before you proceed.

$ dmesg|tail
[ 2806.671425] sd 22:0:0:0: [sdf] Write Protect is off
[ 2806.671430] sd 22:0:0:0: [sdf] Mode Sense: 65 44 09 30
[ 2806.671433] sd 22:0:0:0: [sdf] Assuming drive cache: write through
[ 2806.675283] sd 22:0:0:0: [sdf] 3911616 512-byte hardware sectors: (2.00 GB/1.86 GiB)
[ 2806.675779] sd 22:0:0:0: [sdf] Write Protect is off
[ 2806.675782] sd 22:0:0:0: [sdf] Mode Sense: 65 44 09 30
[ 2806.675783] sd 22:0:0:0: [sdf] Assuming drive cache: write through
[ 2806.675787] sdf: sdf1
[ 2806.677432] sd 22:0:0:0: [sdf] Attached SCSI removable disk
[ 2806.677809] sd 22:0:0:0: Attached scsi generic sg6 type 0


The preceding output reflects the device /dev/sdf is assigned to the newly mounted USB stick. For the rest of this example it will be assumed that the device is /dev/sdf, however this may not be the case with your attempt, so be sure to tail dmesg shortly after installing the USB stick in order to determine the correct device assignment.

Now create a DOS partition and format it. Note that the first step is to delete any existing partitions on the device. This step is only necessary if the device has already been formatted (often a new USB stick will already have been formatted with FAT). Perform the following steps (again note the use of sudo – use whatever method is required by your distro in order to perform these steps with root privilege):

1. Execute fdisk with root privilege:


$ sudo fdisk /dev/sdf
 
The number of cylinders for this disk is set to 2936.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
 
Command (m for help):


2. Print the current partition table:


Command (m for help): p
 
Disk /dev/sdf: 2002 MB, 2002747392 bytes
37 heads, 36 sectors/track, 2936 cylinders
Units = cylinders of 1332 * 512 = 681984 bytes
Disk identifier: 0xeb1326d4
 
Device Boot Start End Blocks Id System
/dev/sdf1 1 2937 1955790 6 FAT16
 
Command (m for help):


3. Delete any existing partitions:


Command (m for help): d
Selected partition 1
 
Command (m for help):


4. Create the new primary partition (this example partitions the entire available space):


Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2936, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2936, default 2936):
Using default value 2936
 
Command (m for help):


5. Set the partition type to W95 FAT32 (this will support partition sizes greater than 2GB):


Command (m for help): t
Selected partition 1
Hex code (type L to list codes): b
Changed system type of partition 1 to b (W95 FAT32)
 
Command (m for help):


6. Set the boot-bit for the partition (this step may be unnecessary as we do this again toward the end using parted):


Command (m for help): a
Partition number (1-4): 1
 
Command (m for help):


7. Verify that the newly created partition is configured correctly before writing to the partition table:


Command (m for help): p
 
Disk /dev/sdf: 2002 MB, 2002747392 bytes
37 heads, 36 sectors/track, 2936 cylinders
Units = cylinders of 1332 * 512 = 681984 bytes
Disk identifier: 0xeb1326d4
 
Device Boot Start End Blocks Id System
/dev/sdf1 * 1 2936 1955358 b W95 FAT32
 
Command (m for help):


8. Write the changes to the partition table:


Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
 
WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.


The USB drive is ready to be formatted as VFAT. It may be necessary to remove and remount the USB drive prior to formatting. It may also be necessary to unmount the newly created partitions that might mount automatically as a result of the kernel caching any previously mounted partitions. In the following example we label the partition g4l-v0.36_0


$ sudo mkfs.vfat -n g4l-v0.36_0 /dev/sdf1
mkfs.vfat 3.0.1 (23 Nov 2008)


It will be necessary to remove and remount the USB stick (again). Depending on your distro, the OS ought to automagically mount the newly formatted USB stick. In the above example we labelled the partition g4l-v0.36_0 which will auto-mount as /media/g4l-v0.36_0.

Copy g4l files to USB drive

Assuming that the latest g4l ISO has been mounted to as g4l in the home directory as instructed above, it is a simple matter to copy all the files to the newly created partition:


$ cp -v ~/g4l/* /media/g4l-v0.36_0/
`/home/mark/g4l/bicode.o3' -> `/media/g4l-v0.36_0/bicode.o3'
`/home/mark/g4l/blank6.cpp' -> `/media/g4l-v0.36_0/blank6.cpp'
`/home/mark/g4l/blank6.exe' -> `/media/g4l-v0.36_0/blank6.exe'
. . .
`/home/mark/g4l/syslinux.cfg' -> `/media/g4l-v0.36_0/syslinux.cfg'
`/home/mark/g4l/test.png' -> `/media/g4l-v0.36_0/test.png'
`/home/mark/g4l/vesamenu.c32' -> `/media/g4l-v0.36_0/vesamenu.c32'


At this point, if you’ve looked at the blog posts referenced in the beginning of this article you will note that we are simply dumping the content of g4l into the root directory of the USB drive. This is for simplicity sake. However, if one wanted to, one could copy the contents of g4l into a subdirectory. However, if this step is taken, then in the following step it will be necessary to edit syslinux.cfg. Notes for what to edit in syslinux.cfg follow in the next section.

Configure bootloader

Once the files have been copied to the USB drive it is necessary to replace the default syslinux.cfg file with the provided isolinux.cfg file as such:

$ sudo cp /media/g4l-v0.36_0/isolinux.cfg /media/g4l-v0.36_0/syslinux.cfg


NOTE:This is the file that has to be modified if g4l was not copied to the root of the USB drive. The following is a snippet of the file (I have edited out much of the detail for sake of brevity):

DEFAULT vesamenu.c32
. . .
LABEL bz32.28
MENU LABEL ^A: bz32.28 386 build 2.6.32.28 01-07-2011
KERNEL bz32.28
APPEND initrd=ramdisk.lzma ramdisk_size=65536 root=/dev/ram0
. . .
LABEL hdt
MENU LABEL ^Z: Hardware Detection Tool 0.36
KERNEL hdt.c32
APPEND modules_pcimap=modules.pcimap pciids=pci.ids memtest=memtest
 
TIMEOUT 600
ONTIMEOUT bz37 initrd=ramdisk.lzma ramdisk_size=65536 root=/dev/ram0


All references to KERNEL, APPEND, DEFAULT (should only be one) and ONTIMEOUT must be modified to include the path to the appropriate files. Also, any directive that makes reference to a file, such as MENU BACKGROUND. The following is the same code example where the g4l files were copied to a directory boot:

DEFAULT boot/vesamenu.c32
. . .
LABEL bz32.28
MENU LABEL ^A: bz32.28 386 build 2.6.32.28 01-07-2011
KERNEL boot/bz32.28
APPEND initrd=boot/ramdisk.lzma ramdisk_size=65536 root=/dev/ram0
. . .
LABEL hdt
MENU LABEL ^Z: Hardware Detection Tool 0.36
KERNEL boot/hdt.c32
APPEND modules_pcimap=boot/modules.pcimap pciids=boot/pci.ids memtest=memtest
 
TIMEOUT 600
ONTIMEOUT bz37 initrd=boot/ramdisk.lzma ramdisk_size=65536 root=/dev/ram0


Create boot image

Once the g4l files have been copied to the USB drive and syslinux.cfg has been modified and moved to the root of the drive, we can then use syslinux to create a bootloader image and copy it to the drive, as follows:

1. First, unmount the USB stick:


$ sudo umount /media/g4l-v0.36_0/


2. Next, generate the bootloader image and copy it to the USB drive. The process for creating the image will copy the image to the USB drive.


$ sudo ~/syslinux-4.03/linux/syslinux /dev/sdf1
/home/mark/syslinux-4.03/linux/syslinux: please specify --install or --update for the future


This is the point at which the other blogs end, leaving one to assume that at this point you can boot from the USB drive. This is incorrect! We still have to copy the MBR (master boot record) to the boot sector of the drive before the bootloader image can be loaded.

Write MBR to USB drive

After following the steps outlined above I was still confounded to discover that my USB drive would not boot. I continued to get an error message telling that it was an invalid boot device. I did some research and found the answer from the Syslinux Wiki^[3]. The key is to copy the mbr.bin file provided with syslinux to the boot sector of the USB drive. This is accomplished as follows:

$ sudo dd conv=notrunc bs=440 count=1 if=~/syslinux-4.03/mbr/mbr.bin of=/dev/sdf
1+0 records in
1+0 records out
440 bytes (440 B) copied, 0.0255058 s, 17.3 kB/s


At this point the USB stick should be ready for use, however to ensure that the partition is indeed flagged as bootable, use parted to set the boot flag. (NOTE: you could set the partition bootable when defining the partition in fdisk, but this is the recommended method from Syslinux Wiki):

$ sudo parted /dev/sdf set 1 boot on
Information: You may need to update /etc/fstab.


That should do it. I use these steps to create multiple USB drives for use at work replicating source “GOLD” images onto new hardware which does not have a CD drive. It’s fast and easy and very reliable. If I have omitted something, or if someone has any suggestions, feel free to leave a comment.</p

References:

1. IT Resource Center forums – bootable USB stick with G4L
2. Radified Community Forum: Bootable USB stick with G4L
3. Syslinux Wiki – How to Create a Bootable USB: For Linux

Brief summary of RAID

RAID, an acronym for redundant array of independent disks, is a technology that provides increased storage reliability through redundancy, combining multiple low-cost, less-reliable disk drives components into a logical unit where all drives in the array are interdependent.^[1]

There are a number of RAID levels which are described in detail at http://en.wikipedia.org/wiki/RAID#Standard_levels. For the purposes of this article I will be focusing on RAID 1 with some references to RAID 0.

Types of RAID

It’s important to note that there are essentiall three types of RAID:

1. Hardware
2. Software
3. FakeRAID

In the case of hardware RAID, all the RAID functionality is handled by the hardware and does not require additional software components as the RAID drive(s) will be exposed to the underlying OS as standard devices.

Software RAID is quite different from hardware RAID. Software RAID is handled by the OS which has a number of implications in terms of availability and performance – not to mention management.

FakeRAID is partial hardware RAID without the features and functions that a true hardware RAID controller offers. This type of RAID has be come increasingly available as more and more motherboards come equipt with BIOS-RAID controls which permit the creation of a RAID array without the ability to manager it (beyound it’s creation and deletion).

FakeRAID is generally supported by Windows OS with the use of additional drivers and software RAID running on the OS. FakeRAID is supported on Linux via dmraid which permits dual-boot between Windows OS and Linux OS whilst maintaining a consistant RAID array.

More on Software RAID

There are essentially two types of software RAID for Linux:

1. dmraid (device-mapper)
2. mdadm (multiple disk administrator)

In the Linux kernel, the device-mapper serves as a generic framework to map one block device onto another. It forms the foundation of LVM2 and EVMS, software RAIDs, dm-crypt disk encryption, and offers additional features such as file-system snapshots.

Device-mapper works by processing data passed in from a virtual block device, that it itself provides, and then passing the resultant data on to another block device.^[2]

mdadm is a Linux utility by Neil Brown that is used to manage software RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software.

mdadm derives its name from the “md” (multiple disk) device nodes it “adm”inisters or manages.^[3]

Which Soft-RAID to use?

There are a numbre of fundamental differences between dmraid and mdadm that I won’t attempt to get into here. However, the decision for which software RAID to choose comes down to one question: Will this PC be configured for dual-boot between Linux and Windows? If the answer is no, then mdadm is the solution to choose.

I have discovered that dmraid does not offer the same level of functionality – particularly when it comes to managing the RAID in cases where the RAID has become degraded or a drive has to be replaced. Early version of dmraid (including the version installed with Ubuntu 9.04) can not rebuild a degraded array. This means that even after replacing a defective drive, the RAID will remain degraded.

On the other hand, mdadm is a fully-functioning software RAID solution which provides all the management tools required to manage all aspects of the array.

Given just these few differences, it’s clear that mdadm is the better software RAID solution for a dedicate Linux PC. However, what if during the installation of the OS dmraid was selected? Can dmraid be converted to use mdadm and therefore permit leverage of all the management tools mdadm provides? Fortunately, the answer is a qualified yes! The qualification assumes that the existing RAID is 1 (and not 0).

Convertion dmraid to mdadm

Overview

For the sake of this article I will make a few assumption:

1. System was installed with RAID 1.
2. At least one disk in the array is fully functioning.
3. At least one disk is available to create the new RAID 1 array
4. The operator has some knowledge of what they are doing!

Essentially the steps that will be covered as as follows:

1. Break dmraid.
2. Disable FakeRAID in BIOS.
3. Create a “broken” RAID 1 array with mdadm.
4. Replicate data from “broken” dmraid array to newly created “broken” mdadm array.
5. Configure system to boot from newly created “broken” mdadm array.
6. Add “broken” dmraid disk into mdadm array.
7. Rebuild mdadm array.

I have borrowed heavily from http://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID for this document. Please refer directly to this document with questions concerning consistency of this document or other issues not covered here.

!!! WARNING !!!

At this point I want to be sure I state clearly that if this document is to be used by someone other than me that your understanding of hardware and the Linux operating system is better than novice! I will not be held responsible for loss of data, bricked hardware, or anything else related to the steps I’m laying out here! You have been warned!

Getting Started: Breaking dmraid!

In order to begin the existing dmraid must be broken and the system must be able to boot from one of the two drives without RAID support. This is a two step process involving erasing the dmraid metadata from the drives making up the array and then removing dmraid – in the process rebuilding the linux kernel such that is does not expect a dmraid device.

To be safe, boot the system as usual and set the run level to 1. This should be done from a tty terminal and not from an open terminal window from the GUI. Use <Ctrl><Alt><F1> to open tty1 then log in and set the run level. This should to be done either as root or with root privileages via sudo.

# telinit 1

This should generate the Recovery Menu from which you can select Root – Drop to root shell prompt. After providing the root user password the system will be running in single-user mode (no multiple ttys) and logged in as root.

To break dmraid it will be necessary to erase the metadate stored on the disks that make up the array. Once the metadata has been erased the system will no longer be bootable via RAID. Therefore, it will be necessary to edit a few files in order to ensure the system remains bootable.

First, however, break the dmraid with:
dmraid -E -r /dev/sd[ab]
NOTE: this assumes the array was built with /dev/sda and /dev/sdb. Your system may vary.

Once the metadata has been removed the drives will no longer be recognized by the BIOS RAID controller as apart of the FakeRAID array. It will be necessary to tell GRUB where the root file system is. It will also be necessary to tell mount where to find the various file system partitions.

Edit /etc/fstab and substitute the appropriate devices for the dmraid-mapped devices. Example:
# vim /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
#
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda5 during installation
#UUID=e9eeafc1-691a-4904-8032-2cc6c75bc175 / ext4 noatime,errors=remount-ro 0 1
/dev/sda2 / ext4 noatime,errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
#UUID=c4a18ea5-336e-42a0-8da8-26f6f3d98d48 /boot ext3 noatime 0 2
/dev/sda1 /boot ext3 noatime 0 2
# swap was on /dev/sda6 during installation
#UUID=64571446-2b40-4d09-9fe0-82d262ebce14 none swap sw 0 0
/dev/sda3 none swap sw 0 0
#/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0

NOTE: You can certainly substitute the UUID for the device, but since this is only a temporary change until the mdadm RAID is complete, there is no point in doing this – just make the changes to the devices and push on.

Add a new entry to /boot/grub/menu.lst which will boot using the device map:
# vim /boot/grub/menu.lst
title Ubuntu 9.04, kernel 2.6.28-19-generic (w/o RAID)
root (hd0,0)
kernel /vmlinuz-2.6.28-19-generic root=/dev/sda2 ro
initrd /initrd.img-2.6.28-19-generic

The above code should be added to the section of menu.lst which describes all the other boot options. Note that the root directive should point to (hdx,y) and the kernel directive root option should point to the physical device defined for root.

Lastly, remove dmraid. Doing so will ensure the correct kernel is in place for the next boot.
# aptitude remove --purge dmraid

Disable FakeRAID in BIOS

At this point boot the system. During the boot be sure to modify the BIOS and disable RAID. I chose to configure SATA as AHCI and set the boot priority to boot from HDD:P0. Other BIOS may have other settings.

If all went well the system should now boot without dmraid to /dev/sda.

Create “broken” RAID 1 array with mdadm.

Once it has been established that the system is bootable without dmraid, install mdadm. This step could have been done eariler without harm to the system. Just be sure mdadm has been installed before proceeding with the following actions.

The first thing that will have to be done is to partition the disk not currently in use (/dev/sdb in this case) for use with mdadm as raid-away partitions. Assuming no physical changes will be made to the layout of the partitions, this simply requires changing the type of the partitions and formatting them with the appropriate filesystem.

Begin by changing the partition type for each of the partitions which will be members of the array to Linux raid autodetect which is type fd.
# sudo fdisk /dev/sdb

Command (m for help):
Repeat these steps for all appropriate devices and write the results. The system will warn you that the new table will not be used until the system has booted. This is expected. Do not boot at this time.

Next, create the single-disk RAID-1 array. Note the “missing” keyword is specified as one of our devices. We are going to fill this missing device later.

# mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm: array /dev/md0 started.


Note: If the above command causes mdadm to say “no such device /dev/sdb2″, then reboot, and run the command again.

If you want to use Grub 0.97 (default in the Ubuntu Linux 9.04) on RAID 1, you need to specify an older version of metadata than the default. Add the option “–metadata=0.90″ to the above command. Otherwise Grub will respond with “Filesystem type unknown, partition type 0xfd” and refuse to install. This is supposedly not necessary with Grub 2.

# mdadm --create /dev/md0 --metadata=0.90 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm: array /dev/md0 started.


Make sure the array has been created correctly by checking /proc/mdstat:
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6] [raid10]
md0 : active raid1 sdb2[1]
40064 blocks [2/1] [_U]

unused devices:

The devices are intact, however in a degraded state. (Because it’s missing half the array!)

References

1. http://en.wikipedia.org/wiki/RAID
2. http://en.wikipedia.org/wiki/Device_mapper
3. http://en.wikipedia.org/wiki/Mdadm

This example assumes a file named: SMCCAPCON_HMBN_20100106.img

$ parted /home/ftp/img/SMCCAPCON_HMBN_20100106.img unit B print
WARNING: You are not superuser. Watch out for permissions.
Warning: Unable to open /home/ftp/img/SMCCAPCON_HMBN_20100106.img read-write
(Permission denied). /home/ftp/img/SMCCAPCON_HMBN_20100106.img has been opened
read-only.
Warning: Unable to open /home/ftp/img/SMCCAPCON_HMBN_20100106.img read-write
(Permission denied). /home/ftp/img/SMCCAPCON_HMBN_20100106.img has been opened
read-only.
Model: (file)
Disk /home/ftp/img/SMCCAPCON_HMBN_20100106.img: 4098834432B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 32256B 3857656319B 3857624064B primary ext3 boot
2 3857656320B 4096189439B 238533120B extended
5 3857688576B 4096189439B 238500864B logical linux-swap


Note that this image contains an entire disk image which has been formatted with a primary partition (#1) which contains root and an extended partition (#5) which contains the swap file.

The next step is to mount the root partition as a loop-back device to a local path. For this example create a mount point at /tmp/mnt (this mount point can of course be anywhere on your file system). Since the image contains both a root partition and the swap partition, it’s critical to use the offset of the root partition as a parameter for the mount command. The offset for root in this example is 32256B.

$ mkdir /tmp/mnt
$ sudo mount -o loop,offset=32256B /home/ftp/img/SMCCAPCON_HMBN_20100106.img /tmp/mnt
The partition is now mounted on /tmp/mnt. To unmount the partition, simply use umount as usual.
$ umount /tmp/mnt


NOTE: If it happens that the image is compressed using lzop, simply uncompress the image first, then take the same actions above. To uncompress a lzop file, use lzop.

$ lzop -x SMCCAPCON_HMBN_20100106.lzo
The resulting file for this example will be SMCCAPCON_HMBN_20100106, not SMCCAPCON_HMBN_20100106.img.
To verify the file type, use the file command.
$ file SMCCAPCON_HMBN_20100106.lzo
SMC_CAPCON_PLM_2010-01-21.lzo: lzop compressed data - version 1.020, LZO1X-1, os: Unix

$ file SMCCAPCON_HMBN_20100106
SMC_CAPCON_PLM_2010-01-21: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 7534422 sectors; partition 2: ID=0x5, starthead 0,
startsector 7534485, 337365 sectors

Recently I ran into a problem with my iPod not automounting. I found this curious as I had been using this same iPod on this same hardware for the last 8 months without issue. Furthermore, I have not made any significant changes to either the iPod nor the laptop I was attempting to connect it to. I am currently running Ubuntu 9.04 on my laptop and the latest iPod software (sorry, don’t know the release, but it was current as of February 27, 2010).

The problem was compounded by the fact that the error message I had been receiving was curious and to my mind misleading. I was getting an error when I’d plug in my iPod:

Cannot mount volume.
Unable to mount the volume 'SILVER_NANO'
Details: mount: /dev/sdb2 already mounted or /media/SILVER_NANO busy


I checked the most obvious: was there something already mounted to /media/SILVER_NANO? No.

When looking at dmesg I found the following relevant output:

[ 150.479730] sd 2:0:0:0: [sdb] Attached SCSI removable disk
[ 150.479889] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 151.240118] sd 2:0:0:0: [sdb] Bad block number requested
[ 151.240136] device-mapper: multipath: Failing path 8:16.
[ 151.240218] Buffer I/O error on device dm-5, logical block 0
[ 151.283296] Buffer I/O error on device dm-5, logical block 1
[ 151.283326] Buffer I/O error on device dm-5, logical block 2
[ 151.283342] Buffer I/O error on device dm-5, logical block 3
[ 151.283361] Buffer I/O error on device dm-5, logical block 4
[ 151.283376] Buffer I/O error on device dm-5, logical block 5
[ 151.283393] Buffer I/O error on device dm-5, logical block 6
[ 151.283408] Buffer I/O error on device dm-5, logical block 7
[ 151.283424] Buffer I/O error on device dm-5, logical block 0
[ 151.283439] Buffer I/O error on device dm-5, logical block 1
[ 151.376242] sd 2:0:0:0: [sdb] Bad block number requested
[ 151.376251] device-mapper: multipath: Failing path 8:16.
[ 151.488138] usb 1-4: reset high speed USB device using ehci_hcd and address 2


In addition to the above messages from dmesg, if I wait long enough the system apparently makes another attempt to mount the device, which also fails. The results from dmesg are as follows”

[ 147.564051] usb 1-4: reset high speed USB device using ehci_hcd and address 2
[ 484.738852] usb 1-4: USB disconnect, address 2
[ 487.764058] usb 1-4: new high speed USB device using ehci_hcd and address 3
[ 487.943593] usb 1-4: configuration #1 chosen from 2 choices
[ 487.993399] scsi3 : SCSI emulation for USB Mass Storage devices
[ 488.005421] usb-storage: device found at 3
[ 488.005429] usb-storage: waiting for device to settle before scanning
[ 493.004381] usb-storage: device scan complete
[ 493.007958] scsi 3:0:0:0: Direct-Access Apple iPod 1.62 PQ: 0 ANSI: 0
[ 493.065093] sd 3:0:0:0: [sdb] 991232 2048-byte hardware sectors: (2.03 GB/1.89 GiB)
[ 493.066165] sd 3:0:0:0: [sdb] Write Protect is off
[ 493.066174] sd 3:0:0:0: [sdb] Mode Sense: 68 00 00 08
[ 493.066182] sd 3:0:0:0: [sdb] Assuming drive cache: write through
[ 493.070541] sd 3:0:0:0: [sdb] 991232 2048-byte hardware sectors: (2.03 GB/1.89 GiB)
[ 493.074162] sd 3:0:0:0: [sdb] Write Protect is off
[ 493.074173] sd 3:0:0:0: [sdb] Mode Sense: 68 00 00 08
[ 493.074180] sd 3:0:0:0: [sdb] Assuming drive cache: write through
[ 493.074192] sdb: sdb1 sdb2
[ 493.078087] sd 3:0:0:0: [sdb] Attached SCSI removable disk
[ 493.078247] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 493.912289] sd 3:0:0:0: [sdb] Bad block number requested
[ 493.912308] device-mapper: multipath: Failing path 8:16.
[ 493.912337] __ratelimit: 100 callbacks suppressed
[ 493.912345] Buffer I/O error on device dm-5, logical block 1
[ 493.912356] Buffer I/O error on device dm-5, logical block 2
[ 493.912366] Buffer I/O error on device dm-5, logical block 3
[ 493.912377] Buffer I/O error on device dm-5, logical block 4
[ 493.912387] Buffer I/O error on device dm-5, logical block 5
[ 493.912397] Buffer I/O error on device dm-5, logical block 6
[ 493.912407] Buffer I/O error on device dm-5, logical block 7
[ 493.912470] Buffer I/O error on device dm-5, logical block 0
[ 493.912497] Buffer I/O error on device dm-5, logical block 0
[ 493.912512] Buffer I/O error on device dm-5, logical block 1

While doing some research and finding nothing specific to my issue, I decided it was time to research multipath specifically, as this is what dmesg is reporting as having trouble. I found the following when I listed the devices mapped by multipath:

$ sudo multipath -ll
2000a27001921f121dm-3 Apple ,iPod
[size=1.9G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
\_ 2:0:0:0 sdb 8:16 [failed][ready]


Suffice to say I’d attempted a number of things before digging into multipath, including attempting to remove and reinstall usb_storage, all without success.

Then I read the manual (RTFM!) for multipath and saw that there is an option to flush all unused multipath device maps (-F). I plugged in my iPod, ran multipath -F and then used my panel applet (Disk Mount) to mount it and viola! my iPod is functioning again.

$ sudo multipath -F
[sudo] password for mark:
libdevmapper: libdm-common.c(374): Removed /dev/mapper/2000a27001921f121-part2
libdevmapper: libdm-common.c(374): Removed /dev/mapper/2000a27001921f121-part1


CONCLUSION: What is the source of this problem? Evidently I recently installed multipath-tools to explore Ubuntu’s support for MPIO devices. Upon removal of this package I find that my iPod automounts just fine.

So, what does this mean? I don’t know exactly, except that having multipath-tools installed on a machine that will be used to work with an iPod is not a good combination.

As such, we have a number of server running Ubuntu 8.04 LTS and 9.04 LTS. Canonical has seen fit to distribute these distribution with GRUB 0.97. And though GRUB2 is available, it has been decided by the company not to install GRUB2 at this time.

This, coupled with a recent change in the look of the company colors, has cause a problem with ‘branding’ the GRUB menu. It seems that GRUB 0.97 does not support the COLOR attribute if the SPLASHIMAGE attribute is in use. The result of which left me with an untenable problem: when I load the new company SPLASHIMAGE which is red lettering on a white background, I was unable to change the color of the text, which is also white by default (thankfully with black shading, so it’s at least visible).

While researching this issue, I came across bug #16190 filed with the GRUB project (https://savannah.gnu.org/bugs/?16190). In it, a helpful user (adrian15) wrote:

Non-standard grub such as the one from Ubuntu is not supported officially by Grub.

However I am going to help you. The color command when having a background image does not work as you have seen.

Use the following commands instead:

background RRGGBB
foreground RRGGBB
adrian15

I followed adrian15′s suggestion, knowing full-well that this is not an officially supported feature of GRUB. My first attempts failed to yield any results. No matter what color combination I defined it was not reflected in the GRUB menu. I continued to explore this option though, as I was certain that had adrian15 been wrong it would have been noted in the bug.

I found while editing the menu interactively I could in fact set the foreground and background colors to values of my choosing. I concluded then that it must be the order in which these settings appeared in /boot/grub/menu.lst that was the problem. Sure enough, by moving the foreground and background setting after the SPLASHIMAGE setting I was able to achieve my desired results!

In summery, if you are using GRUB 0.97 (I don’t care why) and you have need to customize your GRUB menu with a splash image of your own design and you want to change the color of the text, you can do so with the unofficial settings: FOREGROUND and BACKGROUND. Just remember that you must first specify the SPLASHIMAGE, otherwise the color choice will not appear to take effect.

ex:
...
splashimage=(hd0,0)/boot/grub/images/my_splash_image.xpm.gz
foreground ff0000
background 000000
...
This example will set the foreground color of the text to red and the background color of the text to black.

I have added a comment to the now-closed bug entry so that others may benefit. I must stress however that this is an unofficially feature of an obsolete version of GRUB! Do *NOT* expect to get support from the GRUB team or me if this does not work for you!

With the introduction of upstart as a replacement for inittab in ubuntu there are different steps to making qingy work and the qingy (and other) sites suggest. First, install qingy (this part hasn’t changed!). I installed from repos, but you can download and install from the qingy home page (http://qingy.sourceforge.net/). the repos will place the quingy binary in /usr/sbin. Next, edit /etc/event.d/tty1. Comment out the statement # exec /sbin/getty 38400 tty1 and add a new statement exec /usr/sbin/qingy tty1. This will alter tty1 from loading getty to qingy. NOTE: Many how-to’s suggest replacing getty with qingy for all but one tty. I do not feel this is necessary, as I’m merely using qingy as a replacement for gdm and I only need one instance of this.

Next, disable your desktop manager. I’ve stated that I use gdm so if you are using kde or xdm (or something else entirely) you may have to figure out the best way to do this. The easiest way to disable gdm is by using bum (boot-up manager GUI). This will likely have to be installed using your favorite installation method. I use sudo aptitude install bum from the command line. Once running, locate GNOME Display Manager and un-check it. If you click apply the GUI session will end! You have been warned!

At this point you should be able to boot your machine and find that you have a nifty splash screen with options to start different sessions, among them will be text-console.

Note: you will likely want to change some of the default configuration for qingy. For instance, by default qingy will launch your GUI session in the open tty (tty1 in this case). If, like me, you are used to seeing the GUI on tt7, you have to change /etc/qingy/settings x_server_tty = unused_tty. There are many other settings that can be adjusted in the setting files as well.

Also, you can find a nice selection of themes for qingy on freshmeat (http://themes.freshmeat.net)

*** HEADS UP ***
Now, this is where things got a little frustrating for me. When I logged into a gnome session, I discovered that my audio was not initializing. I could not playback audio or work with audio in any way. I did not get any specific errors: dmesg, /var/log/syslog and /var/log/messages did not reveal any obvious errors. I tested audio from a text-console and found that I could playback mp3 files using mpg321 or ximp3 (both console based mp3 players), but only if I run these apps as sudo.

This gave me the vital clue that helped to solve the problem with gnome. It turns out that my user account was not a member of the audio group that is installed when ubuntu was installed. It’s apparently not necessary when you use gdm as your desktop manager. By adding my user to the audio group I was able to playback audio from text-console without the need of sudo *and* when I logged into my gnome session audio was working!

In addition to audio not working, nm-applet (the panel NetworkManager icon control) is also not working.  This item took a little more effort to resolve, as it’s not a simple matter of adding a user account to a group.  The way I resolved this might not be the correct method, so please do your own research! However, I found that if I modified a number of files in /etc/dbus-1/system.d/ I could make it work.  Specifically I modified the following files:

1. /etc/dbus-1/system.d/ConsoleKit.conf
2. /etc/dbus-1/system.d/NetworkManager.conf
3. /etc/dbus-1/system.d/nm-applet.conf
4. /etc/dbus-1/system.d/nm-avahi-autoipd.conf
5. /etc/dbus-1/system.d/ nm-dhcp-client.conf
6. /etv/dbus-1/system.d/nm-dispatcher.conf
7. /etc/dbus-1/system.d/ nm-system-settings.conf

For each of these files I copied the policy user section “root” and changed the user to my user account.

TO REPEAT: This may *not* be the correct way to do this, but in the end it worked.

Another issue I had was with the use of encfs which I used for directory encryption. I found that until I added my user account to the fuse group I was unable to mount my encrypted directories.

I suspect that there are other issues as well with this approach.

If you have any suggestions or comments regarding this method vs. some other method, please post here or provide a link to the suggestion.

Redirect /index.html http://$server/$path/index.html

Where $server and $path represent the server and path to redirect.

This does not appear to be the case with Tomcat, or if it is, I can’t seem to find the right file to update in order to make this change. However, I did find a really good tip from another on how to override the default home page (source sited below).

Essentially when Tomcat is installed it’s configured to look for index.html, index.htm, and index.jsp in $TOMCAT_HOME/webapps/ROOT. (which for my system is /usr/local/tomcat/webapps/ROOT). The index.jsp file located there is the administrator welcome page that is displayed by default. Simply creating an index.html page will override the jsp as the html is loaded before the jsp according to the configuration file located in $TOMCAT_HOME/conf/web.xml. The files and their order is defined in <welcome-file-list>.

The index.html needs to be quite simple and should look as follows:

<html>
  <head>
    <meta http-equiv="refresh" content="0;URL=http[s]://$server/$path/index.html">
  </head>
  <body>
  </body>
</html>

This to me does not appear to be the ideal solution, as it forces the server to perform another round trip, which is inefficient particularly if you are redirecting to the same server but to a different application. However, until I can find a more proper solution such as the apache2 solution, this will have to be what I use.

Source: Tomcat Wiki

• Follow Gentoo Handbook through 4.d. FUTURE: Add notes regarding flushing drive with /dev/urandom before encrypting.
• Be sure to emerge cryptsetup. (NOTE: this should not be necessary as gentoo 2008.0 includes luks support).
• Load necessary modules:
modprobe dm-crypt # required
modprobe dm-mod # required
modprobe serpent # optional
modprobe blowfish # optional
modprobe aes # optional but required for aes-cbc-essiv
• Encrypt the swap partition and mount it. NOTE: Encytping with random passkey – does not require passkey to mount:
  cryptsetup -c aes-cbc-essiv:sha256 -d /dev/urandom create swap /dev/sda2
mkswap /dev/mapper/swap
swapon /dev/mapper/swap
• Encrypt the root partition. NOTE: This example assumes a keyword/passphrase. FUTURE: Add examples for keyfile encryption:
  cryptsetup -y --cipher serpent-cbc-essiv:sha256 --key-size 256 luksFormat /dev/sda3
• Open newly encrypted partition:
  cryptsetup openLuks /dev/sda3 root
• Create the file system on the logical root. This example assumes ext3. FUTURE: Add notes concerning the user of alternate file system:
  mke2fs -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/root -L root
• Create the file system for /boot. This example assumes ext2 (as you needn’t journal a boot partition). FUTURE: Add notes concerning the use of alternate file system:
  mke2fs /dev/sda1 -L boot
• Mount the file system for use during the remainder of install:
  mount /dev/mapper/root /mnt/gentoo
• Return to the Gentoo Handbook from step 5 to 7.d (genkernel)
• Build the kernel with genkernel
  cd /usr/src/linux
genkernel --luks --menuconfig all
  The key here is that you configure your kernel for ramdisk support and luks support FUTURE: add details.
• Continue with Gentoo Handbook from 7.e till 8.a
• Modify /etc/fstab to mount root from /dev/mapper/root. fstab will look as follows:
  

  /dev/sda1	/boot	ext2	defaults,noatime	1	2
  /dev/mapper/swap	none	swap	sw	0	0
  /dev/mapper/root	/	ext3	noatime	0	1

• Modify /etc/conf.d/dmcrypt to mount swap during boot:
  swap=swap
source='/dev/sda2'
• Continue with Gentoo Handbook from 8.b till 10.b.
• Configure GRUB to use genkernel:
  

  title	Gentoo Linux 2.6.25-r8 (genkernel)
  root	(hd0,0)
  kernel	/boot/kernel-genkernel-x86-2.6.25-gentoo-r8 root=/dev/ram0 init=/linuxrc ramdisk=8192 crypt_root=/dev/sda3 vga=791
  initrd	/boot/initramfs-genkernel-x86-2.6.25-gentoo-r8

  NOTE:kernel name may be different,be sure to use the correct initramfs and kernel.

Unbelievably that’s it! After days and days of trying to create my own initramfs and writing my own linuxrc files it really comes down to passing using genkernel to build the kernel and initramfs with luks support by passing genkernel –luks! (well, and the rest).

More to come later.

Citation:
http://gentoo-wiki.com/SECURITY_System_Encryption_DM-Crypt_with_LUKS#Preparing_the_disk
http://gentoo-wiki.com/SECURITY_Encrypting_Root_Filesystem_with_DM-Crypt_with_LUKS#Using_initramfs_and_busybox
http://gentoo-wiki.com/SECURITY_Encrypting_Root_Filesystem_with_DM-Crypt
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=1

Here is a handy one for when you have to chroot into your system because you omitted something and it doesn’t boot: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=6

Solution:
Tunnel traffic through an ssh connection and forward the traffic from some other port (assuming you have mysql running on your localhost on port 3306) to port 3306.

How?

Simple: Open a new shell and create the tunnel on the local machine to the remote server as follows:

ssh -L $localport:127.0.0.1:3306 -l $remoteuserid $remoteserveraddress

• $localport can be any available port, include 3306, but only if mysql is *not* running on the local machine.
• $remoteuserid is the user id used to connect to the remote server.
• $remoteserveraddress is the address of the remote server.

Once the tunnel is established, leave the shell open. You might get timed out depending on your server configuration, so you might need to have some application running on the remote server to maintain the connection.

Now launch mysql-query-browser, set the Server Hostname to 127.0.0.1 and the port to $localport. That’s it!