Brain Dump

A place to store my random thoughts and anything else I might find useful.

Archive for May, 2012

Repairing Broken Arch Linux

Posted by mzanfardino on May 24, 2012

Contents

1. Overview
2. Solution
3. Summary

Overview

I recently ran into a situation in which my laptop would not fully boot. I’m running Arch Linux and I’d recently synchronized my packages when low-and-behold at my next boot the kernel could not find the hard disk. This resulted in my system booting to a busybox shell.

Some further discussion of my particular configuration. I am using partition-level encryption along with LVM2 partition management. As such the first thing that has to happen for the system to fully boot is for me to provide my decryption passkey. Since the kernel could not ‘see’ the encrypted partition (or any other partitions for that matter) the boot could progress no further.

Solution

The solution was to boot to a ‘live’ distro (in my case I used BackTrack 5.2), decrypt the encrypted root partition, mount it (along with the boot partition) to an arbitrary mount point on the ‘live’ filesystem, then chroot and repair. The following are a step-by-step transcription of the actions I took:

  • Boot system to ‘live’ CD (BackTrack in this case)
  • Decrypt root filesystem

  • # cryptsetup luksOpen /dev/sda6 crypto
    Enter passphrase for /dev/sda6:**********
    Key slot 0 unlocked.

  • Mount root and boot file systems

  • # mkdir /mnt/arch
    # mount /dev/mapper/vg-lv_root /mnt/arch
    # mount /dev/sda5 /mnt/arch/boot

  • chroot the newly mounted filesystem

  • # cd /mnt/arch
    # mount -t proc proc proc/
    # mount -t sysfs sys sys/
    # mount -o bind /dev dev/
    # chroot . /bin/bash
    [root@bt /]#

  • Bring up network device (eth0 in this case)

  • [root@bt /]# dhcpcd eth0
    dhcpcd[....]: version 5.5.6 starting
    dhcpcd[....]: eth0: sending IPv6 Router Solicitation
    dhcpcd[....]: eth0: rebinding lease of xx.xx.xx.xx
    dhcpcd[....]: eth0: acknowledged xx.xx.xx.xx from xx.xx.xx.yy
    dhcpcd[....]: eth0: leased xx.xx.xx.xx for 86400 second
    dhcpcd[....]: forked to background, child pid ....

  • Refresh all packages and upgrade

  • [root@bt /]# pacman -Syy
    :: Synchronizing package databases...
    core
    extra
    community
    multilib

    [root@bt /]# pacman -Syu
    :: Synchronizing package databases...
    core is up to date
    extra is up to date
    community is up to date
    multilib id up to date
    :: Starting full system upgrade...
    there is nothing to do

  • Reinstall udev and mkinitcpio

  • [root@bt /]# pacman -S udev
    warning: udev-182-4 is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): udev-182-4

    Total Installed Size: 1.62 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading udev [###############] 100%

    [root@bt /]# pacman -S mkinitcpio
    warning: mkinitcpio-0.9.0.2 is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): mkinitcpio-0.9.0.2

    Total Installed Size: 0.25 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading mkinitcpio [###############] 100%

  • Create the initial ramdisk environment

  • [root@bt /]# mkinitcpio -p linux
    ==> Building image from preset: 'default'
    -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
    ==> Starting build: 3.3.7-1ARCH
    -> Parsing hook: [base]
    -> Parsing hook: [udev]
    -> Parsing hook: [autodetect]
    -> Parsing hook: [pata]
    -> Parsing hook: [scsi]
    -> Parsing hook: [sata]
    -> Parsing hook: [keymap]
    -> Parsing hook: [encrypt]
    -> Parsing hook: [lvm2]
    -> Parsing hook: [filesystems]
    -> Parsing hook: [usbinput]
    -> Parsing hook: [fsck]
    Ok.
    ==> Generating module dependencies
    ==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
    ==> Image generation successful

  • Reboot

  • [root@bt /]# exit
    # reboot

Summary

It appears as though the last time the initial ramdisk environment was created the kernel somehow became corrupt. As such the kernel lacked support to recognize and mount hard disk drives and their partitions. By booting to a live distro I was able to mount the encrypted file system, chroot into newly mounted file system, and re-run the processes that likely broke the system to begin with (presuming that they would not break the system again!).

Thanks (as always) to help from ewanm89 (Ewan Marshall) for guiding me in this process!

Posted in linux | 7 Comments »