Brain Dump

A place to store my random thoughts and anything else I might find useful.

Repairing Broken Arch Linux

Posted by mzanfardino on May 24, 2012

Contents

1. Overview
2. Solution
3. Summary

Overview

I recently ran into a situation in which my laptop would not fully boot. I’m running Arch Linux and I’d recently synchronized my packages when low-and-behold at my next boot the kernel could not find the hard disk. This resulted in my system booting to a busybox shell.

Some further discussion of my particular configuration. I am using partition-level encryption along with LVM2 partition management. As such the first thing that has to happen for the system to fully boot is for me to provide my decryption passkey. Since the kernel could not ‘see’ the encrypted partition (or any other partitions for that matter) the boot could progress no further.

Solution

The solution was to boot to a ‘live’ distro (in my case I used BackTrack 5.2), decrypt the encrypted root partition, mount it (along with the boot partition) to an arbitrary mount point on the ‘live’ filesystem, then chroot and repair. The following are a step-by-step transcription of the actions I took:

  • Boot system to ‘live’ CD (BackTrack in this case)
  • Decrypt root filesystem

  • # cryptsetup luksOpen /dev/sda6 crypto
    Enter passphrase for /dev/sda6:**********
    Key slot 0 unlocked.

  • Mount root and boot file systems

  • # mkdir /mnt/arch
    # mount /dev/mapper/vg-lv_root /mnt/arch
    # mount /dev/sda5 /mnt/arch/boot

  • chroot the newly mounted filesystem

  • # cd /mnt/arch
    # mount -t proc proc proc/
    # mount -t sysfs sys sys/
    # mount -o bind /dev dev/
    # chroot . /bin/bash
    [root@bt /]#

  • Bring up network device (eth0 in this case)

  • [root@bt /]# dhcpcd eth0
    dhcpcd[....]: version 5.5.6 starting
    dhcpcd[....]: eth0: sending IPv6 Router Solicitation
    dhcpcd[....]: eth0: rebinding lease of xx.xx.xx.xx
    dhcpcd[....]: eth0: acknowledged xx.xx.xx.xx from xx.xx.xx.yy
    dhcpcd[....]: eth0: leased xx.xx.xx.xx for 86400 second
    dhcpcd[....]: forked to background, child pid ....

  • Refresh all packages and upgrade

  • [root@bt /]# pacman -Syy
    :: Synchronizing package databases...
    core
    extra
    community
    multilib

    [root@bt /]# pacman -Syu
    :: Synchronizing package databases...
    core is up to date
    extra is up to date
    community is up to date
    multilib id up to date
    :: Starting full system upgrade...
    there is nothing to do

  • Reinstall udev and mkinitcpio

  • [root@bt /]# pacman -S udev
    warning: udev-182-4 is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): udev-182-4

    Total Installed Size: 1.62 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading udev [###############] 100%

    [root@bt /]# pacman -S mkinitcpio
    warning: mkinitcpio-0.9.0.2 is up to date -- reinstalling
    resolving dependencies...
    looking for inter-conflicts...

    Targets (1): mkinitcpio-0.9.0.2

    Total Installed Size: 0.25 MiB
    Net Upgrade Size: 0.00 MiB
    Proceed with installation? [Y/n]
    (1/1) checking package integrity
    (1/1) loading package files [###############] 100%
    (1/1) checking for file conflicts [###############] 100%
    (1/1) checking available disk space [###############] 100%
    (1/1) upgrading mkinitcpio [###############] 100%

  • Create the initial ramdisk environment

  • [root@bt /]# mkinitcpio -p linux
    ==> Building image from preset: 'default'
    -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
    ==> Starting build: 3.3.7-1ARCH
    -> Parsing hook: [base]
    -> Parsing hook: [udev]
    -> Parsing hook: [autodetect]
    -> Parsing hook: [pata]
    -> Parsing hook: [scsi]
    -> Parsing hook: [sata]
    -> Parsing hook: [keymap]
    -> Parsing hook: [encrypt]
    -> Parsing hook: [lvm2]
    -> Parsing hook: [filesystems]
    -> Parsing hook: [usbinput]
    -> Parsing hook: [fsck]
    Ok.
    ==> Generating module dependencies
    ==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
    ==> Image generation successful

  • Reboot

  • [root@bt /]# exit
    # reboot

Summary

It appears as though the last time the initial ramdisk environment was created the kernel somehow became corrupt. As such the kernel lacked support to recognize and mount hard disk drives and their partitions. By booting to a live distro I was able to mount the encrypted file system, chroot into newly mounted file system, and re-run the processes that likely broke the system to begin with (presuming that they would not break the system again!).

Thanks (as always) to help from ewanm89 (Ewan Marshall) for guiding me in this process!

Advertisements

7 Responses to “Repairing Broken Arch Linux”

  1. Sebastian said

    Awesome article! You just saved my day (and probably my week). Thanks!!

  2. C said

    I’m so happy that these instructions worked, I’m close to wanting to name my first-born after you.

  3. Greg said

    Thanks! I had a weird problem where, after I would type in my username and password to log in, getty would restart and I’d be back where I started. This fixed it, though I don’t really know what was wrong.

  4. John said

    Awesome work. Is there some similar way to repair only /var when it has gone corrupt on it’s own harddrive? I can mount /var but i get io-error when trying to cd down /var/db and so on.

    • mzanfardino said

      I would think that you would need to run a file system scan – something like e2fsck /dev/sdX – depending on your filesystem and device of course. Beyond that, it would depend on what the file system scan revealed – it might correct the corruption, but if your disk is failing you have a more serious problem.

  5. J said

    Had a hang when doing a pacman update for the kernel – which I suspect was down to a stale nfs mount that wouldn’t clear – got no update output, but on reboot it couldn’t restore.
    Got corrupt packages (evidently just mismatching keys) while doing the -Syu, but can skip them plus udev without an issue, and was able to mkinitcpio by specifying ‘linux39’ as the preset., It recovered it without disastrous side effects. Grub still says Manjaro-3.9.7, but it’s booting into 3.9.10 without a problem, so I imagine the next update will ‘fix’ grubs output.
    Thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: