Brain Dump

A place to store my random thoughts and anything else I might find useful.

Linux: Enabling brightness control for Thinkpad W520

Posted by mzanfardino on February 10, 2015


1. Overview
2. Solution
3. Summary


I have a Lenovo ThinkPad W520 laptop and I’m running Arch Linux. Recently I’ve found that my brightness controls have not been working. Specifically the OSD for my dm (Cinnamon in my case) was reporting the brightness level changing but the screen backlight was not changing. I know this feature worked in the past which tells me that some change/update has ‘broken’ this functionality.

The solution required making two changes: one to grub.cfg and the other to xorg.conf. The change to grub.cfg enables the thinkpad-acpi brightness control events and the change to xorg.conf couples the key events to the acpi.


When researching this problem the first thing I took a look at was the boot log. Given that Arch has moved to systemd (a topic for another post) I used journalctl -b to search for references to ‘thinkpad’. I found the following entries:

Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: ThinkPad ACPI Extras v0.25
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi:
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: ThinkPad BIOS 8BET56WW (1.36 ), EC unknown
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: Lenovo ThinkPad W520, model 4270CTO
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: detected a 16-level brightness capable ThinkPad
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: radio switch found; radios are enabled
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: This ThinkPad has standard ACPI backlight brightness control, supported by the AC
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: Disabling thinkpad-acpi brightness events by default...
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: Standard ACPI backlight interface available, not loading native one
Nov 10 17:13:49 enceladus kernel: thinkpad_acpi: Console audio control enabled, mode: monitor (read only)
Nov 10 17:13:49 enceladus kernel: input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/input/input8

The first thing I noticed was that the kernel was reporting that the thinkpad-acpi brightness events where disable. This lead me to conclude that a kernel flag was necessary in order to enable the thinkpad-acpi brightness events. After some research I found a post on Ubuntu Wiki[1] which suggested that I edit /etc/default/grub and added the following:

GRUB_CMDLINE_LINUX_DEFAULT="acpi_backlight=vendor thinkpad-acpi.brightness_enable=1"

Rebuilt grub.cfg using:

# grub-mkconfig -o /boot/grub/grub.cfg

The same Ubuntu Wiki post suggested a change to /etc/X11/xorg.conf. I changes my ‘Section “Device”‘ to include the following two lines:

BoardName "Quadro 1000M"
Option "RegistryDwords" "EnableBrightnessControl=1"

My new ‘Section “Device”‘ looks like the following:

Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro 1000M"
Option "RegistryDwords" "EnableBrightnessControl=1"

After rebooting I found that in the journal the thinkpad-acpi brightness events was no longer disabled however the brighness controls themselves were still not working correctly. Specifically I noticed that the OSD was no longer reporting a valid percentage value but was not reporting a value from 0 to 16. After a little more research I found a power on Arch Linux Wiki [2] which suggested that rather than using the “acpi_backlight=vendor” kernel flag I should be using the ‘acpi_osi=”!Windows 2012″‘ flag. I changed my grub.cfg linux default to the following:

GRUB_CMDLINE_LINUX_DEFAULT="acpi_osi=\"!Windows 2012\" thinkpad-acpi.brightness_enable=1"

I then rebuild grub.cfg and booted. This time my brightness controls worked as expected and the OSD was reporting the correct values. I was also able to verify that the brightness controls were working for any TTY terminal.

Although this solution worked, I did not like having the reference to “!Windows 2012” in my kernel flags – this is linux for crying out loud! Why do I have to make a Windows reference (despite the fact that evidently Lenovo changed the acpi driver to accommodate Windows). I found a reference to ‘acpi_osi=linux’ in another Arch Linux wiki article[3] so I made one final change to my kernel flag in grub to the following:

GRUB_CMDLINE_LINUX_DEFAULT="acpi_osi=linux thinkpad-acpi.brightness_enable=1"

After rebuilding grub.cfg everything worked as expected.


As a result of a change in the thinkpad-acpi driver (evidently) the default behavior for brightness control is disabled. This requires that the kernel be notified through a flag to enable the brightness events. In addition to enabling the events the kernel needs to be notified that this osi is linux (not Windows). This is handled by a separate flag. Finally, xorg needs to be configured to handle the brightness controls as well. Once both the bootloader and x11 are configured correctly the brightness controls should work.

Please note that I’ve made a number of assumptions. I’m assuming the bootloader is grub – I presume other bootloaders will permit kernel flags to be set. I’m also presuming the use of xorg.conf. If this isn’t your case, please feel free to drop me a comment with how you were able to resolve your individual problem.

[1] Screen brightness control not working on Lenovo T530
[2] Thinkpad T430 backlight control
[3] Intel graphics

Posted in acpi, Arch Linux, grub, kernel, linux, thinkpad-acpi, xorg | Leave a Comment »

Transcoding Video for Android

Posted by mzanfardino on September 17, 2013


1. Overview
2. Solution
3. Summary


Android devices have a limited number of video and audio codecs that are supported natively. I have a collection of videos that I’d like to be able to watch on my Nexus 10, but simply uploading the files does not always work. Many of the files are encoded in mkv or avi containers with mp3 or ac3 audio encoding. These will not play with the native video player and attempts to install a full-featured player such as vlc have been frustrating. The solution is to transcode the video to mp4 encoding with aac audio encoding. Note: there are other compatible video and audio codecs such as 3gpp, but I have found that mp4 and aac are most straight-forward codecs to transcode.


Transcoding can be performed with a number of different tools and a number of different approaches may well be valid. I am documenting here the solution that works for me. I have been using ffmpeg for years to transcode, so naturally this is the tool I turned to. Assuming all the appropriate libraries have been installed, you can use ffmpeg to transcode from mkv or avi with a few simple switches.

The first issue I wanted to address was the size of the files. A number of my files are in excess of 1GB. This is due to the frame-size of the videos: some as large as 1280×720. I clearly don’t need that high resolution for playback on my Nexus 10, so reducing the frame-size to something more manageable has the desired effect of reducing the file size. I prefer video scaled to 720xW (where W is relative to the original scale dimension). Why this particular frame-size? It’s high-enough resolution to maintain a decent quality video; it’s small enough that most videos will down-size to this frame-size (rather than forcing an up-size which can be ugly); and maintains the correct aspect ratio. You can of course substitute any frame-size you like.

The second issue I need to address is audio codecs. Many of the videos I have are encoded using ac3 which is not supported. I chose aac as my audio codec as this is supported by the native video player and it will preserve 5.1 if the source is encoded as such.

The following is the general form of the command I use:

$ ffmpeg -i foo.mkv -c:v libx264 -preset slow -crf 22 -vf scale=[1920,1280,720]:-1 -c:a aac -strict -2 foo.mp4

I wanted to transcode both avi and mkv in one pass. I also wanted to do more than one file at a time, so I constructed the following command to transcode a block of files:

$ for f in *.{avi,mkv}; do ffmpeg -i ${f} -c:v libx264 -preset slow -crf 22 -vf scale=[1920,1280,720]:-1 -c:a aac -strict -2 ${f/${f: -3}/mp4}; done;

NOTE: in the above the ‘scale’ parameter should be only one of the three suggested values. By setting the width to -1 the resulting video will be scaled correctly. One caveat: in some instances the width value returned from -1 will be an odd number which will cause ffmpeg to throw an error. In that case I mere adjust the width to the next highest even integer and rerun the command providing the calculated width rather than the -1.

There are a number of consideration when performs transcoding that I have not touched on. I have tried to address the most common situations I’ve run into relating to: video and audio codec as well as maintaining aspect ratio.

The final step is to transfer the newly transcoded videos to my Nexus. This was done with a simple adb push command (no need to root access).


Android is a wonderful platform with a number of fantastic features. However, given the limited set of video and audio codecs supported, it’s necessary to convert your videos you might otherwise be able to “just play” on your PC. The process I’ve outlined is just one way to achieve this goal. With linux and adb it can be a very easy process to convert your videos to a playable format (while reducing the file size) and move them to your favorite android device.

Posted in android, ffmpeg, linux, transcode, video | Tagged: , , , , | Leave a Comment »

Dual boot UEFI Windows 7 and Ubuntu 12.04 (adapted from

Posted by mzanfardino on November 20, 2012

In an effort to ensure that I do not forget how to perform the following tasks, I am re-posting them here. This is a near-verbatim copy of the instructions provided at

Find where the windows bootloader resides by mounting the efi partition, and searching for bootmgfw.efi. My configuration has Windows installed on four (4) separate devices: sda1, sda2, sda3, and sda4. The GRUB scripts find windows installed on sda3, but this does not boot as it’s not the correct EFI entry. In my case, I found bootmgfw.efi on sda1.

$ mount /dev/sda1 ~/tmp/sda1
$ grub-probe --target=fs_uuid ~/tmp/sda1/EFI/Microsoft/Boot/bootmgfw.efi

Edit grub custom script to add an entry for UEFI boot for Windows 7:

$ vim /etc/grub.d/40_custom
menuentry "Windows x86_64 UEFI-GPT" {
search --fs-uuid --no-floppy --set=root 2431-71DE
chainloader (${root})/efi/Microsoft/Boot/bootmgfw.efi

NOTE: replace the UUID follow ‘–set=root’ with the value returned from grub-probe. Then run update-grub:

$ sudo update-grub

This worked in my case.

The PC I configured for DualBoot is an HP Pavilion pre-installed with Windows 7 Home Premium. I used gparted to resize sda3 to make space before I installed ubuntu. I installed ubuntu from the alternate install CD (64-bit).

Posted in Uncategorized | Leave a Comment »

How-To: Rename/Change User in Linux

Posted by mzanfardino on October 4, 2012


1. Overview
2. Solution
3. Notes
4. Attribution

NOTE: This entry is just a basic overview and should not be considered complete. Changing an existing user can be complicated by application configuration dependencies that may not be changed when the user is changed. Be sure you have a backup of your data before you proceed!


Recently I have found that I have a need to change my user account on a virtual machine (VM) running liunx to a more generic account in order that others may use the virtual machine without having to use my user credentials. I find VM’s extremely handy for portable development and stability, as I don’t always want to expose my development environment to the kinds of changes that often accompany upgrades in my desktop distribution.

To that end, I have found that with a few fairly simple commands I can not only rename my user account, but ensure that group associations are maintained and the home path is updated appropriately. However, as I state at the beginning of this article, none of the following commands will guarantee that installed applications will work without a little ‘tweaking’. In my case, my Netbeans project files needed to be updated with the new home path as I did not define them with relative paths but rather with direct path references. You have been warned (again)


Before beginning, but sure you have logged out of your user account. Do not attempt to run these commands from a shell while logged in as your user! In my case, I booted my machine using the “recovery mode” and dropped to the root-user shell to issue the following commands.

# killall -u old
# id old
# usermod -l new old
# groupmod -n new old
# usermod -d /home/new -m new
# usermod -c “New Real Name” new
# id new

The following is a brief summary of what these commands do. For more information, RTFM (remember, man is your friend)

‘killall -u old’ will kill all processes owned by the user ‘old’ where ‘old’ should be the name of the user to be changed.

‘id old’ will print out the real and effective user and group ID’s for user ‘old’.

‘usermod -l new old’ will change the user name ‘old’ to ‘new’ (where ‘new’ is the new user name to be used). Note that this only changes the users name, not his home directory, groups or any else.

‘groupmod -n new old’ will change the group name ‘old’ to ‘new’. Similar to the previous usermod command, this command will only update the group and nothing else.

‘usermod -d /home/new -m new’ will change the new user’s home folder to /home/new.

‘usermod -c “New Real Name” new’ will change the user’s password file comment field only.

‘id new’ will print out the real and effective user and group ID’s for user ‘new’. This is done so that you can verify that the expected changes have occurred.


These commands should result in changing the user ‘old’ to a new user ‘new’ as well as creating and assigning a new home folder and ensuring that all groups that ‘old’ belonged to are transferred to ‘new’. After logging into the new account, I promptly changed the user password (with passwd) and will encourage the user of this VM to change it to something unique to them and changed my keyring password (using seahorse).

Lastly, I install a few utilities that rely on a local MTA (postfix in my case) and found I had to modify the postfix configuration in order to ensure continued mail receipt. I also run apt-listchanges which is configured to mail to my account the changelogs and news of any packages that are to be changed/updated (I highly recommend installing this feature if you run a debian machine and use apt). I had to reconfigure apt-listchanges in order to update the mail account name to use when sending these notifications. Finally, I had to dig deep into my Netbeans project folder to locate and private.xml in order to update the path assignments for project paths and server paths such as for my instance of apache tomcat.

I can’t be sure that this covers all my specific issues however, and in time may discover other configuration files that will need to be changed. However, from the looks of it, these steps will cover about 90%+ of the changes required.


Thanks to the post I found at and the comment from a user who identifies himself as simply ‘ubuntu’ for the expanded set of commands.

Posted in debian, linux, ubuntu | 1 Comment »

How-To: Remove errant CR (Carriage return, ^M, ‘\r’, 0x0D, 13 in decimal) from text file.

Posted by mzanfardino on July 5, 2012


1. Overview
2. Solution
3. Summary
4. References


There are occasions when one has to work with a text file originating from a DOS/Windows platform on a *nix machine. Opening the file on *nix reveals the addition of a carriage return (^M) appended to the end of each line. This occurs as a result of DOS/Windows utilizing a combination of the CR (Carriage return, ^M, ‘\r’, 0x0D, 13 in decimal) + LF (Line feed, ^L, ‘\n’, 0x0A, 10 in decimal) to indicate a new line while *nix traditionally uses just the LF[1].


Depending on the depth of the problem there are a number of solutions one might implement. When dealing with an individual file I find that vi/m is the right tool for the job. I imagine emac or other editors are equally suited to the task, but as I’m a vi/m guy, this is what works for me:


$ vim file_with_errant_cr
This is a test^M
This is also a test^M
Here again, another test^M
What, more tests?^M
Yeah, but this is the last test.^M

NOTE: the ^v in this context is the Ctrl-v key-combination and ^M is the Ctrl-m key-combination. These are literals; do not type ^v and ^m. When typed the output will look like:



This is a test
This is also a test
Here again, another test
What, more tests?
Yeah, but this is the last test.

That’s it! Save the file and all the errant CR (^M) are gone.

ED: Additional research has revealed (several) additional, easy solutions to this problem. One of the easiest is to issue the following command in vim:


This simple but effective alternative replaces the last character of each line (.$) with nothing (//). No need for ^v/^m key combos. However, it is important to note that this does exactly what I said; so let me restate: it replaces the last character of each line regardless of what that character is! So be sure this is what you want to do before doing it!

When dealing with multiple files it is impractical to have to edit each and every file. In this case I would write a simple bash routine which uses sed to find and replace the errant character and save the files to something else. Let’s take for example a group of files named testn.txt where n is some incremental value:

$ for f in test*.txt; do cat ${f} | sed s/^v^M//g > ${f/.txt/.new}; done;

As with vi/m, the ^v is the Ctrl-v key-combination and ^M is the Ctrl-m key-combination. When typed the output will look like:

$ for f in test*.txt; do cat ${f} | sed s/^M//g > ${f/.txt/.new}; done;

The results will be new files created absent the CR with file extension .new. The original files will be left untouched.


The approaches I’ve taken here were inspired by research I did on the subject and the input I received from Quinn McHenry’s post on Tech Recipes[2]. Quinn also explains clearly that “In UNIX, you can escape a control character by preceeding it with a CONTROL-V”. This was critical to my understanding of how to use sed as well as understanding how vi/m can be used to replace control characters.


  1. Newline: From Wikipedia, the free encyclopedia
  2. Remove ^M characters at end of lines in vi by Quinn McHenry

Posted in Uncategorized | Leave a Comment »

Repairing Broken Arch Linux

Posted by mzanfardino on May 24, 2012


1. Overview
2. Solution
3. Summary


I recently ran into a situation in which my laptop would not fully boot. I’m running Arch Linux and I’d recently synchronized my packages when low-and-behold at my next boot the kernel could not find the hard disk. This resulted in my system booting to a busybox shell.

Some further discussion of my particular configuration. I am using partition-level encryption along with LVM2 partition management. As such the first thing that has to happen for the system to fully boot is for me to provide my decryption passkey. Since the kernel could not ‘see’ the encrypted partition (or any other partitions for that matter) the boot could progress no further.


The solution was to boot to a ‘live’ distro (in my case I used BackTrack 5.2), decrypt the encrypted root partition, mount it (along with the boot partition) to an arbitrary mount point on the ‘live’ filesystem, then chroot and repair. The following are a step-by-step transcription of the actions I took:

  • Boot system to ‘live’ CD (BackTrack in this case)
  • Decrypt root filesystem

  • # cryptsetup luksOpen /dev/sda6 crypto
    Enter passphrase for /dev/sda6:**********
    Key slot 0 unlocked.

  • Mount root and boot file systems

  • # mkdir /mnt/arch
    # mount /dev/mapper/vg-lv_root /mnt/arch
    # mount /dev/sda5 /mnt/arch/boot

  • chroot the newly mounted filesystem

  • # cd /mnt/arch
    # mount -t proc proc proc/
    # mount -t sysfs sys sys/
    # mount -o bind /dev dev/
    # chroot . /bin/bash
    [root@bt /]#

  • Bring up network device (eth0 in this case)

  • [root@bt /]# dhcpcd eth0
    dhcpcd[....]: version 5.5.6 starting
    dhcpcd[....]: eth0: sending IPv6 Router Solicitation
    dhcpcd[....]: eth0: rebinding lease of xx.xx.xx.xx
    dhcpcd[....]: eth0: acknowledged xx.xx.xx.xx from xx.xx.xx.yy
    dhcpcd[....]: eth0: leased xx.xx.xx.xx for 86400 second
    dhcpcd[....]: forked to background, child pid ....

  • Refresh all packages and upgrade

  • [root@bt /]# pacman -Syy
    :: Synchronizing package databases...

    [root@bt /]# pacman -Syu
    :: Synchronizing package databases...
    core is up to date
    extra is up to date
    community is up to date
    multilib id up to date
    :: Starting full system upgrade...
    there is nothing to do

  • Reinstall udev and mkinitcpio

  • [root@bt /]# pacman -S udev
    warning: udev-182-4 is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): udev-182-4

    Total Installed Size: 1.62 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading udev [###############] 100%

    [root@bt /]# pacman -S mkinitcpio
    warning: mkinitcpio- is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): mkinitcpio-

    Total Installed Size: 0.25 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading mkinitcpio [###############] 100%

  • Create the initial ramdisk environment

  • [root@bt /]# mkinitcpio -p linux
    ==> Building image from preset: 'default'
    -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
    ==> Starting build: 3.3.7-1ARCH
    -> Parsing hook: [base]
    -> Parsing hook: [udev]
    -> Parsing hook: [autodetect]
    -> Parsing hook: [pata]
    -> Parsing hook: [scsi]
    -> Parsing hook: [sata]
    -> Parsing hook: [keymap]
    -> Parsing hook: [encrypt]
    -> Parsing hook: [lvm2]
    -> Parsing hook: [filesystems]
    -> Parsing hook: [usbinput]
    -> Parsing hook: [fsck]
    ==> Generating module dependencies
    ==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
    ==> Image generation successful

  • Reboot

  • [root@bt /]# exit
    # reboot


It appears as though the last time the initial ramdisk environment was created the kernel somehow became corrupt. As such the kernel lacked support to recognize and mount hard disk drives and their partitions. By booting to a live distro I was able to mount the encrypted file system, chroot into newly mounted file system, and re-run the processes that likely broke the system to begin with (presuming that they would not break the system again!).

Thanks (as always) to help from ewanm89 (Ewan Marshall) for guiding me in this process!

Posted in linux | 7 Comments »

Problems with ThinkPad W520 handing during udev

Posted by mzanfardino on April 6, 2012


1. Overview
2. Solution
3. Summary


I’m going to keep this short. In summary, I just bought a ThinkPad W520 and found that when I attempt to install a number of the latest linux distros they all hang intermittently during boot. I decided to troubleshoot this using Arch Linux as my test platform, as I have no trouble booting from the netinst CD nor installing the OS.  However, after the OS is installed I have found that the system hangs more than 50% of the time during “Waiting to process udev events”.  Naturally I assumed the problem lay in udev. As of yesterday (April 5, 2012) the version of udev installed from core is 181-5. I opted to try udev from testing (udev 181-9). This did not improve things.


With some help from folks in #udev I was able to disable asynchronous processing and enable debugging from the kernel options in grub-legacy.  This helped me to identify a key clue: it never seemed to hang at the same spot.  I reasoned that if it was in fact udev that was failing, and given that I disabled asynchronous processing it seems highly unlikely that the problem was in fact with udev.

This led me to think of hardware, such as RAM or CPU as the source of the problem.  However, after several successful passes with memtest86+ 4.20 I was convinced that there was no problem with RAM.  This left the CPU.

After scouring the web for days and fighting with this issue for nearly a week, I finally came across a forum post for Fedora that suggested perhaps the problem lay with Intel Virtualization Technology being enabled.

And there it was: I’d turned on Intel-VT when I first booted the machine and never ran it with Intel-VT turned off. After turning Intel-VT off I have been able to successfully boot to Arch 100% of the time!


As udev has nothing (directly) to do with Intel-VT, it’s highly unlikely that udev is the problem. It’s more likely udev is loading a module (it has been suggested to me by a friend of mine Ewen Marshall that  kvm is the likely suspect) which is the real culprit. I will be performing tests this weekend in an effort to isolate the offending module and hopefully then open a bug with the developers of the module in an effort to help solve the problem.

ED – I found this recently which suggests that perhaps it’s localized to VT-d (and not the entire VT) in combination with discrete video.  I happen to be running with discrete video enabled in BIOS as well, and haven’t (yet) tried to boot with VT enabled and VT-d disabled.  I will try that in addition to my other tests this weekend and will report back later.

ED – I can confirm that the issue appears to be related to the VT-d.  I have enabled VT and left VT-d disabled.  I will begin further research as time permits.

Posted in Arch Linux, linux | 3 Comments »

How to mirror installed apps on Debian-based linux distros

Posted by mzanfardino on November 22, 2011

I will flesh this out with comments later. Suffice to say, if one needs to mirror the applications installed on one Debian-based linux distro (such as Ubuntu) to another (also Debian-based, preferably running the save release) one merely needs to generate a list of those apps installed on host A and configure host B to install the same apps. This is done using dpkg and apt-get.

Generate the list by logging into host A:

# dpkg --get-selections \* > apps.lst

Copy apps.list to host B and execute the following:

# dpkg --set-selections < apps.lst
# apt-get -u dselect-upgrade

If there are apps which need to be installed you will see the familiar apt-get/aptitude interface displaying the apps to be installed and a prompt to confirm. Once confirmed, the apps will be installed and both machines should be mirrors (from an application standpoint) of each other.

Posted in Uncategorized | Tagged: | 1 Comment »

How To: Create Bootable g4l USB Flash Drive

Posted by mzanfardino on February 18, 2011

This ‘How To’ covers in detail how to create a bootable USB flash drive with the imaging software g4l installed. g4l (or ghost4linux) is a free and open source hard disk and partition imaging/cloning tool. The purpose is to eliminate the need for a CD in cases where a CD drive may not be available.

This article presumes one is running Linux (in my case Ubuntu 10.04 though in principal this should work for pretty much any Linux distro running a 2.6 kernel). I will describe the process for formatting the USB flash drive, mounting an ISO file to the local file system in order to copy files to the USB drive, and installing a bootloader and making the USB drive bootable.

While searching the web for similar solutions, I came across a few other blogs[1][2] that describe the procedure, only to discover that they lacked the critical component for how to make the drive bootable. Therefore, I’m going to make my own attempt to cover this topic in a more structured, comprehensive fashion.

Basic Outline

1. Prerequisites
2. Create a FAT partition on target USB drive
3. Copy g4l files to USB drive
4. Configure bootloader
5. Create boot image
6. Write MBR to USB drive
7. References


There are a few software requirements that must be met before proceeding. Firstly, be sure to have a copy of the latest version of the g4l ISO file (you can download version 0.36 here.

Secondly, it will be necessary to have a bootloader available. In this article, I will be using syslinux. syslinux is a lightweight bootloader for MS-DOS FAT filesystems and will be used to generate the bootloader image. You can download the latest version here. I understand that GRUB or possibly LILO could be used, but for this article I will stick to syslinux

Lastly, be sure to have parted installed. parted is a command line tool for manipulating partition tables. This is useful for creating space for new operating systems, re-organizing disk usage, copying data on hard disks and disk imaging. This may already be installed with your distro, but if not, be sure to install it either from the package repository or from source.

Once the required software has been downloaded and/or installed, begin by opening a terminal window and mounting the g4l ISO to a local mount point. In the following example the mount point is the directory g4l created in the users home directory. NOTE: unless your system is configured to permit your user account to mount filesystems it will be necessary to have root privilege to mount the ISO. In the case of Ubuntu I use sudo, however it may be necessary to su root depending on your distro (your mileage may vary!)

$ sudo mount -o loop ~/downloads/iso/utilities/g4l-v0.36.iso ~/g4l
Next, expand the syslinux tar file. In the following example I expand the file into my home directory:

$ tar xjvf ~/downloads/iso/utilities/syslinux-4.03.tar.bz2
. . .

We are now ready to proceed with the real work!

Create a FAT partition on target USB drive

To being, mount the USB drive and observe the output from dmesg to determine what device identifier is assigned. Depending on your distro and if the drive has already been formatted it might auto-mount. If so, be sure to unmount the device before you proceed.

$ dmesg|tail
[ 2806.671425] sd 22:0:0:0: [sdf] Write Protect is off
[ 2806.671430] sd 22:0:0:0: [sdf] Mode Sense: 65 44 09 30
[ 2806.671433] sd 22:0:0:0: [sdf] Assuming drive cache: write through
[ 2806.675283] sd 22:0:0:0: [sdf] 3911616 512-byte hardware sectors: (2.00 GB/1.86 GiB)
[ 2806.675779] sd 22:0:0:0: [sdf] Write Protect is off
[ 2806.675782] sd 22:0:0:0: [sdf] Mode Sense: 65 44 09 30
[ 2806.675783] sd 22:0:0:0: [sdf] Assuming drive cache: write through
[ 2806.675787] sdf: sdf1
[ 2806.677432] sd 22:0:0:0: [sdf] Attached SCSI removable disk
[ 2806.677809] sd 22:0:0:0: Attached scsi generic sg6 type 0

The preceding output reflects the device /dev/sdf is assigned to the newly mounted USB stick. For the rest of this example it will be assumed that the device is /dev/sdf, however this may not be the case with your attempt, so be sure to tail dmesg shortly after installing the USB stick in order to determine the correct device assignment.

Now create a DOS partition and format it. Note that the first step is to delete any existing partitions on the device. This step is only necessary if the device has already been formatted (often a new USB stick will already have been formatted with FAT). Perform the following steps (again note the use of sudo – use whatever method is required by your distro in order to perform these steps with root privilege):

  1. Execute fdisk with root privilege:

  2. $ sudo fdisk /dev/sdf
    The number of cylinders for this disk is set to 2936.
    There is nothing wrong with that, but this is larger than 1024,
    and could in certain setups cause problems with:
    1) software that runs at boot time (e.g., old versions of LILO)
    2) booting and partitioning software from other OSs
    (e.g., DOS FDISK, OS/2 FDISK)
    Command (m for help):

  3. Print the current partition table:

  4. Command (m for help): p
    Disk /dev/sdf: 2002 MB, 2002747392 bytes
    37 heads, 36 sectors/track, 2936 cylinders
    Units = cylinders of 1332 * 512 = 681984 bytes
    Disk identifier: 0xeb1326d4
    Device Boot Start End Blocks Id System
    /dev/sdf1 1 2937 1955790 6 FAT16
    Command (m for help):

  5. Delete any existing partitions:

  6. Command (m for help): d
    Selected partition 1
    Command (m for help):

  7. Create the new primary partition (this example partitions the entire available space):

  8. Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    Partition number (1-4): 1
    First cylinder (1-2936, default 1):
    Using default value 1
    Last cylinder, +cylinders or +size{K,M,G} (1-2936, default 2936):
    Using default value 2936
    Command (m for help):

  9. Set the partition type to W95 FAT32 (this will support partition sizes greater than 2GB):

  10. Command (m for help): t
    Selected partition 1
    Hex code (type L to list codes): b
    Changed system type of partition 1 to b (W95 FAT32)
    Command (m for help):

  11. Set the boot-bit for the partition (this step may be unnecessary as we do this again toward the end using parted):

  12. Command (m for help): a
    Partition number (1-4): 1
    Command (m for help):

  13. Verify that the newly created partition is configured correctly before writing to the partition table:

  14. Command (m for help): p
    Disk /dev/sdf: 2002 MB, 2002747392 bytes
    37 heads, 36 sectors/track, 2936 cylinders
    Units = cylinders of 1332 * 512 = 681984 bytes
    Disk identifier: 0xeb1326d4
    Device Boot Start End Blocks Id System
    /dev/sdf1 * 1 2936 1955358 b W95 FAT32
    Command (m for help):

  15. Write the changes to the partition table:

  16. Command (m for help): w
    The partition table has been altered!
    Calling ioctl() to re-read partition table.
    WARNING: If you have created or modified any DOS 6.x
    partitions, please see the fdisk manual page for additional
    Syncing disks.

    The USB drive is ready to be formatted as VFAT. It may be necessary to remove and remount the USB drive prior to formatting. It may also be necessary to unmount the newly created partitions that might mount automatically as a result of the kernel caching any previously mounted partitions. In the following example we label the partition g4l-v0.36_0

    $ sudo mkfs.vfat -n g4l-v0.36_0 /dev/sdf1
    mkfs.vfat 3.0.1 (23 Nov 2008)

It will be necessary to remove and remount the USB stick (again). Depending on your distro, the OS ought to automagically mount the newly formatted USB stick. In the above example we labelled the partition g4l-v0.36_0 which will auto-mount as /media/g4l-v0.36_0.

Copy g4l files to USB drive

Assuming that the latest g4l ISO has been mounted to as g4l in the home directory as instructed above, it is a simple matter to copy all the files to the newly created partition:

$ cp -v ~/g4l/* /media/g4l-v0.36_0/
`/home/mark/g4l/bicode.o3' -> `/media/g4l-v0.36_0/bicode.o3'
`/home/mark/g4l/blank6.cpp' -> `/media/g4l-v0.36_0/blank6.cpp'
`/home/mark/g4l/blank6.exe' -> `/media/g4l-v0.36_0/blank6.exe'
. . .
`/home/mark/g4l/syslinux.cfg' -> `/media/g4l-v0.36_0/syslinux.cfg'
`/home/mark/g4l/test.png' -> `/media/g4l-v0.36_0/test.png'
`/home/mark/g4l/vesamenu.c32' -> `/media/g4l-v0.36_0/vesamenu.c32'

At this point, if you’ve looked at the blog posts referenced in the beginning of this article you will note that we are simply dumping the content of g4l into the root directory of the USB drive. This is for simplicity sake. However, if one wanted to, one could copy the contents of g4l into a subdirectory. However, if this step is taken, then in the following step it will be necessary to edit syslinux.cfg. Notes for what to edit in syslinux.cfg follow in the next section.

Configure bootloader

Once the files have been copied to the USB drive it is necessary to replace the default syslinux.cfg file with the provided isolinux.cfg file as such:

$ sudo cp /media/g4l-v0.36_0/isolinux.cfg /media/g4l-v0.36_0/syslinux.cfg

NOTE:This is the file that has to be modified if g4l was not copied to the root of the USB drive. The following is a snippet of the file (I have edited out much of the detail for sake of brevity):

DEFAULT vesamenu.c32
. . .
LABEL bz32.28
MENU LABEL ^A: bz32.28 386 build 01-07-2011
KERNEL bz32.28
APPEND initrd=ramdisk.lzma ramdisk_size=65536 root=/dev/ram0
. . .
MENU LABEL ^Z: Hardware Detection Tool 0.36
KERNEL hdt.c32
APPEND modules_pcimap=modules.pcimap pciids=pci.ids memtest=memtest
ONTIMEOUT bz37 initrd=ramdisk.lzma ramdisk_size=65536 root=/dev/ram0

All references to KERNEL, APPEND, DEFAULT (should only be one) and ONTIMEOUT must be modified to include the path to the appropriate files. Also, any directive that makes reference to a file, such as MENU BACKGROUND. The following is the same code example where the g4l files were copied to a directory boot:

DEFAULT boot/vesamenu.c32
. . .
LABEL bz32.28
MENU LABEL ^A: bz32.28 386 build 01-07-2011
KERNEL boot/bz32.28
APPEND initrd=boot/ramdisk.lzma ramdisk_size=65536 root=/dev/ram0
. . .
MENU LABEL ^Z: Hardware Detection Tool 0.36
KERNEL boot/hdt.c32
APPEND modules_pcimap=boot/modules.pcimap pciids=boot/pci.ids memtest=memtest
ONTIMEOUT bz37 initrd=boot/ramdisk.lzma ramdisk_size=65536 root=/dev/ram0

Create boot image

Once the g4l files have been copied to the USB drive and syslinux.cfg has been modified and moved to the root of the drive, we can then use syslinux to create a bootloader image and copy it to the drive, as follows:

  1. First, unmount the USB stick:

  2. $ sudo umount /media/g4l-v0.36_0/

  3. Next, generate the bootloader image and copy it to the USB drive. The process for creating the image will copy the image to the USB drive.

  4. $ sudo ~/syslinux-4.03/linux/syslinux /dev/sdf1
    /home/mark/syslinux-4.03/linux/syslinux: please specify --install or --update for the future

This is the point at which the other blogs end, leaving one to assume that at this point you can boot from the USB drive. This is incorrect! We still have to copy the MBR (master boot record) to the boot sector of the drive before the bootloader image can be loaded.

Write MBR to USB drive

After following the steps outlined above I was still confounded to discover that my USB drive would not boot. I continued to get an error message telling that it was an invalid boot device. I did some research and found the answer from the Syslinux Wiki[3]. The key is to copy the mbr.bin file provided with syslinux to the boot sector of the USB drive. This is accomplished as follows:

$ sudo dd conv=notrunc bs=440 count=1 if=~/syslinux-4.03/mbr/mbr.bin of=/dev/sdf
1+0 records in
1+0 records out
440 bytes (440 B) copied, 0.0255058 s, 17.3 kB/s

At this point the USB stick should be ready for use, however to ensure that the partition is indeed flagged as bootable, use parted to set the boot flag. (NOTE: you could set the partition bootable when defining the partition in fdisk, but this is the recommended method from Syslinux Wiki):

$ sudo parted /dev/sdf set 1 boot on
Information: You may need to update /etc/fstab.

That should do it. I use these steps to create multiple USB drives for use at work replicating source “GOLD” images onto new hardware which does not have a CD drive. It’s fast and easy and very reliable. If I have omitted something, or if someone has any suggestions, feel free to leave a comment.</p


  1. IT Resource Center forums – bootable USB stick with G4L
  2. Radified Community Forum: Bootable USB stick with G4L
  3. Syslinux Wiki – How to Create a Bootable USB: For Linux

Posted in linux | 3 Comments »

Thoughts on RAID for Linux

Posted by mzanfardino on October 18, 2010

The following article is for my own personal benefit, however comments are welcome, as the resulting document is limited to my personal experience with soft-RAID on Ubuntu 9.04.

Brief summary of RAID

RAID, an acronym for redundant array of independent disks, is a technology that provides increased storage reliability through redundancy, combining multiple low-cost, less-reliable disk drives components into a logical unit where all drives in the array are interdependent.[1]

There are a number of RAID levels which are described in detail at For the purposes of this article I will be focusing on RAID 1 with some references to RAID 0.

Types of RAID

It’s important to note that there are essentiall three types of RAID:

  1. Hardware
  2. Software
  3. FakeRAID

In the case of hardware RAID, all the RAID functionality is handled by the hardware and does not require additional software components as the RAID drive(s) will be exposed to the underlying OS as standard devices.

Software RAID is quite different from hardware RAID. Software RAID is handled by the OS which has a number of implications in terms of availability and performance – not to mention management.

FakeRAID is partial hardware RAID without the features and functions that a true hardware RAID controller offers. This type of RAID has be come increasingly available as more and more motherboards come equipt with BIOS-RAID controls which permit the creation of a RAID array without the ability to manager it (beyound it’s creation and deletion).

FakeRAID is generally supported by Windows OS with the use of additional drivers and software RAID running on the OS. FakeRAID is supported on Linux via dmraid which permits dual-boot between Windows OS and Linux OS whilst maintaining a consistant RAID array.

More on Software RAID

There are essentially two types of software RAID for Linux:

  1. dmraid (device-mapper)
  2. mdadm (multiple disk administrator)

In the Linux kernel, the device-mapper serves as a generic framework to map one block device onto another. It forms the foundation of LVM2 and EVMS, software RAIDs, dm-crypt disk encryption, and offers additional features such as file-system snapshots.

Device-mapper works by processing data passed in from a virtual block device, that it itself provides, and then passing the resultant data on to another block device.[2]

mdadm is a Linux utility by Neil Brown that is used to manage software RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software.

mdadm derives its name from the “md” (multiple disk) device nodes it “adm”inisters or manages.[3]

Which Soft-RAID to use?

There are a numbre of fundamental differences between dmraid and mdadm that I won’t attempt to get into here. However, the decision for which software RAID to choose comes down to one question: Will this PC be configured for dual-boot between Linux and Windows? If the answer is no, then mdadm is the solution to choose.

I have discovered that dmraid does not offer the same level of functionality – particularly when it comes to managing the RAID in cases where the RAID has become degraded or a drive has to be replaced. Early version of dmraid (including the version installed with Ubuntu 9.04) can not rebuild a degraded array. This means that even after replacing a defective drive, the RAID will remain degraded.

On the other hand, mdadm is a fully-functioning software RAID solution which provides all the management tools required to manage all aspects of the array.

Given just these few differences, it’s clear that mdadm is the better software RAID solution for a dedicate Linux PC. However, what if during the installation of the OS dmraid was selected? Can dmraid be converted to use mdadm and therefore permit leverage of all the management tools mdadm provides? Fortunately, the answer is a qualified yes! The qualification assumes that the existing RAID is 1 (and not 0).

Convertion dmraid to mdadm


For the sake of this article I will make a few assumption:

  1. System was installed with RAID 1.
  2. At least one disk in the array is fully functioning.
  3. At least one disk is available to create the new RAID 1 array
  4. The operator has some knowledge of what they are doing!

Essentially the steps that will be covered as as follows:

  1. Break dmraid.
  2. Disable FakeRAID in BIOS.
  3. Create a “broken” RAID 1 array with mdadm.
  4. Replicate data from “broken” dmraid array to newly created “broken” mdadm array.
  5. Configure system to boot from newly created “broken” mdadm array.
  6. Add “broken” dmraid disk into mdadm array.
  7. Rebuild mdadm array.

I have borrowed heavily from for this document. Please refer directly to this document with questions concerning consistency of this document or other issues not covered here.

!!! WARNING !!!

At this point I want to be sure I state clearly that if this document is to be used by someone other than me that your understanding of hardware and the Linux operating system is better than novice! I will not be held responsible for loss of data, bricked hardware, or anything else related to the steps I’m laying out here! You have been warned!

Getting Started: Breaking dmraid!

In order to begin the existing dmraid must be broken and the system must be able to boot from one of the two drives without RAID support. This is a two step process involving erasing the dmraid metadata from the drives making up the array and then removing dmraid – in the process rebuilding the linux kernel such that is does not expect a dmraid device.

To be safe, boot the system as usual and set the run level to 1. This should be done from a tty terminal and not from an open terminal window from the GUI. Use <Ctrl><Alt><F1> to open tty1 then log in and set the run level. This should to be done either as root or with root privileages via sudo.

# telinit 1

This should generate the Recovery Menu from which you can select Root – Drop to root shell prompt. After providing the root user password the system will be running in single-user mode (no multiple ttys) and logged in as root.

To break dmraid it will be necessary to erase the metadate stored on the disks that make up the array. Once the metadata has been erased the system will no longer be bootable via RAID. Therefore, it will be necessary to edit a few files in order to ensure the system remains bootable.

First, however, break the dmraid with:
dmraid -E -r /dev/sd[ab]
NOTE: this assumes the array was built with /dev/sda and /dev/sdb. Your system may vary.

Once the metadata has been removed the drives will no longer be recognized by the BIOS RAID controller as apart of the FakeRAID array. It will be necessary to tell GRUB where the root file system is. It will also be necessary to tell mount where to find the various file system partitions.

Edit /etc/fstab and substitute the appropriate devices for the dmraid-mapped devices. Example:
# vim /etc/fstab
# /etc/fstab: static file system information.
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda5 during installation
#UUID=e9eeafc1-691a-4904-8032-2cc6c75bc175 / ext4 noatime,errors=remount-ro 0 1
/dev/sda2 / ext4 noatime,errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
#UUID=c4a18ea5-336e-42a0-8da8-26f6f3d98d48 /boot ext3 noatime 0 2
/dev/sda1 /boot ext3 noatime 0 2
# swap was on /dev/sda6 during installation
#UUID=64571446-2b40-4d09-9fe0-82d262ebce14 none swap sw 0 0
/dev/sda3 none swap sw 0 0
#/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0

NOTE: You can certainly substitute the UUID for the device, but since this is only a temporary change until the mdadm RAID is complete, there is no point in doing this – just make the changes to the devices and push on.

Add a new entry to /boot/grub/menu.lst which will boot using the device map:
# vim /boot/grub/menu.lst
title Ubuntu 9.04, kernel 2.6.28-19-generic (w/o RAID)
root (hd0,0)
kernel /vmlinuz-2.6.28-19-generic root=/dev/sda2 ro
initrd /initrd.img-2.6.28-19-generic

The above code should be added to the section of menu.lst which describes all the other boot options. Note that the root directive should point to (hdx,y) and the kernel directive root option should point to the physical device defined for root.

Lastly, remove dmraid. Doing so will ensure the correct kernel is in place for the next boot.
# aptitude remove --purge dmraid

Disable FakeRAID in BIOS

At this point boot the system. During the boot be sure to modify the BIOS and disable RAID. I chose to configure SATA as AHCI and set the boot priority to boot from HDD:P0. Other BIOS may have other settings.

If all went well the system should now boot without dmraid to /dev/sda.

Create “broken” RAID 1 array with mdadm.

Once it has been established that the system is bootable without dmraid, install mdadm. This step could have been done eariler without harm to the system. Just be sure mdadm has been installed before proceeding with the following actions.

The first thing that will have to be done is to partition the disk not currently in use (/dev/sdb in this case) for use with mdadm as raid-away partitions. Assuming no physical changes will be made to the layout of the partitions, this simply requires changing the type of the partitions and formatting them with the appropriate filesystem.

Begin by changing the partition type for each of the partitions which will be members of the array to Linux raid autodetect which is type fd.
# sudo fdisk /dev/sdb

The number of cylinders for this disk is set to 77825.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs

Command (m for help): t
Partition number (1-7): 1
Hex code (type L to list codes): fd

Command (m for help):
Repeat these steps for all appropriate devices and write the results. The system will warn you that the new table will not be used until the system has booted. This is expected. Do not boot at this time.

Next, create the single-disk RAID-1 array. Note the “missing” keyword is specified as one of our devices. We are going to fill this missing device later.

# mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm: array /dev/md0 started.

Note: If the above command causes mdadm to say “no such device /dev/sdb2”, then reboot, and run the command again.

If you want to use Grub 0.97 (default in the Ubuntu Linux 9.04) on RAID 1, you need to specify an older version of metadata than the default. Add the option “–metadata=0.90” to the above command. Otherwise Grub will respond with “Filesystem type unknown, partition type 0xfd” and refuse to install. This is supposedly not necessary with Grub 2.

# mdadm --create /dev/md0 --metadata=0.90 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm: array /dev/md0 started.

Make sure the array has been created correctly by checking /proc/mdstat:
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6] [raid10]
md0 : active raid1 sdb2[1]
40064 blocks [2/1] [_U]

unused devices:

The devices are intact, however in a degraded state. (Because it’s missing half the array!)



Posted in commentary, kubuntu, linux, ubuntu | Leave a Comment »