Repairing Broken Arch Linux

Contents
1. Overview
2. Solution
3. Summary

Overview

I recently ran into a situation in which my laptop would not fully boot. I’m running Arch Linux and I’d recently synchronized my packages when low-and-behold at my next boot the kernel could not find the hard disk. This resulted in my system booting to a busybox shell.

Some further discussion of my particular configuration. I am using partition-level encryption along with LVM2 partition management. As such the first thing that has to happen for the system to fully boot is for me to provide my decryption passkey. Since the kernel could not ‘see’ the encrypted partition (or any other partitions for that matter) the boot could progress no further.

Solution

The solution was to boot to a ‘live’ distro (in my case I used BackTrack 5.2), decrypt the encrypted root partition, mount it (along with the boot partition) to an arbitrary mount point on the ‘live’ filesystem, then chroot and repair. The following are a step-by-step transcription of the actions I took:

• Boot system to ‘live’ CD (BackTrack in this case)
• Decrypt root filesystem


# cryptsetup luksOpen /dev/sda6 crypto
Enter passphrase for /dev/sda6:**********
Key slot 0 unlocked.


• Mount root and boot file systems


# mkdir /mnt/arch
# mount /dev/mapper/vg-lv_root /mnt/arch
# mount /dev/sda5 /mnt/arch/boot


• chroot the newly mounted filesystem


# cd /mnt/arch
# mount -t proc proc proc/
# mount -t sysfs sys sys/
# mount -o bind /dev dev/
# chroot . /bin/bash
[root@bt /]#


• Bring up network device (eth0 in this case)


[root@bt /]# dhcpcd eth0
dhcpcd[....]: version 5.5.6 starting
dhcpcd[....]: eth0: sending IPv6 Router Solicitation
dhcpcd[....]: eth0: rebinding lease of xx.xx.xx.xx
dhcpcd[....]: eth0: acknowledged xx.xx.xx.xx from xx.xx.xx.yy
dhcpcd[....]: eth0: leased xx.xx.xx.xx for 86400 second
dhcpcd[....]: forked to background, child pid ....


• Refresh all packages and upgrade


[root@bt /]# pacman -Syy
:: Synchronizing package databases...
core
extra
community
multilib

[root@bt /]# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
multilib id up to date
:: Starting full system upgrade...
there is nothing to do


• Reinstall udev and mkinitcpio


[root@bt /]# pacman -S udev
warning: udev-182-4 is up to date -- reinstalling
resolving dependencies...
looking for inter-conflicts...

Targets (1): udev-182-4

Total Installed Size: 1.62 MiB
Net Upgrade Size: 0.00 MiB
Proceed with installation? [Y/n]
(1/1) checking package integrity
(1/1) loading package files [###############] 100%
(1/1) checking for file conflicts [###############] 100%
(1/1) checking available disk space [###############] 100%
(1/1) upgrading udev [###############] 100%

[root@bt /]# pacman -S mkinitcpio
warning: mkinitcpio-0.9.0.2 is up to date -- reinstalling
resolving dependencies...
looking for inter-conflicts...

Targets (1): mkinitcpio-0.9.0.2

Total Installed Size: 0.25 MiB
Net Upgrade Size: 0.00 MiB
Proceed with installation? [Y/n]
(1/1) checking package integrity
(1/1) loading package files [###############] 100%
(1/1) checking for file conflicts [###############] 100%
(1/1) checking available disk space [###############] 100%
(1/1) upgrading mkinitcpio [###############] 100%


• Create the initial ramdisk environment


[root@bt /]# mkinitcpio -p linux
==> Building image from preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 3.3.7-1ARCH
-> Parsing hook: [base]
-> Parsing hook: [udev]
-> Parsing hook: [autodetect]
-> Parsing hook: [pata]
-> Parsing hook: [scsi]
-> Parsing hook: [sata]
-> Parsing hook: [keymap]
-> Parsing hook: [encrypt]
-> Parsing hook: [lvm2]
-> Parsing hook: [filesystems]
-> Parsing hook: [usbinput]
-> Parsing hook: [fsck]
Ok.
==> Generating module dependencies
==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful


• Reboot


[root@bt /]# exit
# reboot


Summary

It appears as though the last time the initial ramdisk environment was created the kernel somehow became corrupt. As such the kernel lacked support to recognize and mount hard disk drives and their partitions. By booting to a live distro I was able to mount the encrypted file system, chroot into newly mounted file system, and re-run the processes that likely broke the system to begin with (presuming that they would not break the system again!).

Thanks (as always) to help from ewanm89 (Ewan Marshall) for guiding me in this process!