Fixing a Broken Linux System with chroot

Image result for linux images

Chroot is a tool, recognized by skilled Linux users, which can be used to rescue or repair broken Linux systems. The reliability of modern system installers which shroud their actions has made it virtually unknown to the average user. However, with the “Do it yourself” distros such as Arch or Gentoo Linux things are easier to muck up. Installation errors can occur through ignorance, neglect, or merely an oversight.

Linux users generally consider Arch or Gentoo to be difficult distributions to install and use; virtually impossible for a Linux novice. A common issue with any Linux installation such as these which requires that you manually create or edit configuration files is faulty entries. Simple misspellings, omitted entries, or punctuation errors can leave a new system crippled or worse unbootable.

For instance, one of the first times I installed Arch I added the following line to my hosts file:

127.0.0.1	vbox-arch.localdomain	vbox-arch

instead of…

127.0.1.1	vbox-arch.localdomain	vbox-arch

Notice the error? The simple difference in the IP address caused my internet connection to fail. Since the system was still bootable I simply opened the /etc/hosts file again and made the change. However, when your new system will not start or the terminal is not responding (which happens when the localization is not correctly set up, done that, too) the following steps may allow you to make things right without drastic measures.

While it may be possible to chroot into a Fedora, openSUSE, or Ubuntu systems using a Recovery/Rescue or Troubleshooting mode of their installation image, this post will simply consider Arch and Gentoo.

Chroot, what is it and why do I care?

Chroot is the process of booting a good system image and then changing the /root partition to the installed root directory. This provides access to the new root directory as if it was the one that was booted. In fact, the chrooted directory cannot “see” the original directory. All further commands and processes are now performed within this isolated environment or “jail”. A chroot can also be performed in a working system simply to change to a different /root. An explanation of chroot that will help with the understanding of this post can be found here.

The two different chroot procedures covered here are a manual command-line process vs the “automated’ arch-chroot script. The arch-chroot procedure uses a bash script which does a lot of the work with one command and sets up a nice chroot environment.

The following examples are from Virtualbox guest installations, but they are not much different from running the systems on a PC. You must use a bootable image (either ISO or USB) to get your system to the point where you can run both chroot procedures.

The tale of two chroots

I consider Gentoo to be a more challenging distro of the two to manage. Maybe because it “breaks” a lot, but I have learned the most about Linux struggling with Gentoo over the years. Arch, however, seems to be much more popular and for this reason, I will discuss how to get back into a broken, unbootable, or simply misconfigured Arch Linux setup.

Both of these systems use a chroot from the installation media to perform the real install and configuring of each new setup. Because of this, it is actually easier to repair them then a distro which hides the installation process from you. This work, though, is all from the command line, no GUI, so be prepared.

Arch provides a bash script that does the same work as the manual Gentoo process explained here. Both require a little more advanced knowledge of Linux, but most users should easily follow these directions.

Get the Boot image…

First, you will need to get the appropriate installation (boot) media. The architecture of the image must be of the same as that of the system into which you will chroot, for example, amd64 or x86. Some install media have both on the same image.

Once you get either installation (or boot) ISO or USB, you simply boot the image on the device which contains the damaged system. Since I used Virtualbox installs I set the guest’s optical drive contents to access the ISO.

Know your disk layout…

Since the purpose of chroot is to mount a valid directory as /root you must know what disk and/or partition you want to mount. With either of the following ways to chroot, you must have this information. The disk and partition data can be gotten from several programs, such as fdisk, sfdisk, cfdisk, to name a few.

If you do not know or cannot recall the hard drive or partition layout of the damaged system you can use the lsblk -f command to list them and their filesystem type. (the following is shown from the Gentoo LiveCD prompt):

Output of the lsblk -f command (List block devices and filesystem types) more output to the right is cut off

You can also use fdisk for more information.

Output of the fdisk /dev/sda -l command showing the known partitions and filesystems

As shown it looks like sda4 is the /root directory, sda2 is the /boot directory and sda3 is the swap partition. Of course, this is a somewhat simple layout. This post is not meant to cover the topic of complex disk layouts or strategies. This illustration simply demonstrates a way to get the information that you will need in both of the following examples.

Now you have all the needed information to chroot into the above described system.

Arch-Chroot

Before running the arch-chroot script you must first mount the partition/drives to which you will be changing and accessing, such as /root and /boot. If you get an error that the mount point does not exist then you must create it using the mkdir command.

Arch Linux installation image boot type selector

From the boot type selection list chose the “Install medium…”, choice number one as previously shown. After Arch boots running fdisk -l tells us that there are only 2 partitions which we need to be concerned with, the /root and the /swap.

Only 2 partitions in the example Arch system…

First, we need to turn on the swap partition like so…

root@archiso ~ #swapon /dev/sda2

Now we mount the one and only other partition to the /mnt directory. This step could include one or two more commands depending on the disk layout which exists.

root@archiso ~ #mount /dev/sda1 /mnt

Finally, we run one command referencing the “mounted” /root partition to access the installed system

root@archiso ~ #arch-chroot /mnt

Now you are running the installed system from which you can run pacman or manually edit config files. Remember that you are logged on as root. To exit the chroot and return to the booted image’s filesystem simply enter exit

root@archiso /]# exit

To completely shut down the system (and give you time to safely remove the boot medium) use “shutdown -h now”. That is as about as simple as it gets. Gentoo must be chrooted manually using several more steps. The manual process can also be used on Arch if you want to add to your work.

The Manual Process…Gentoo

To manually chroot a Gentoo installation we mount will the two partitions listed in the output of the disk /dev/sda -l command shown at the beginning of this post. These will be mounted to the /mnt/gentoo and /mnt/gentoo/boot directories. Additionally, there will be several virtual file system directories to mount. These initial steps could include more commands depending on the disk layout which exists. First, the existing swap partition needs to be turned on…

livecd ~ # swapon /dev/sda3

Next, we mount the /root and /boot partitions. Remember in the case of this Linux installation there are 4 partitions. We will mount to the /mnt/gentoo directory because that is how the Gentoo installation guide does it. You can follow your own directory structure. If you get an error that the mount point does not exist then you must create using the mkdir command.

livecd ~ # mount /dev/sda4 /mnt/gentoo
livecd ~ # mount /dev/sda2 /mnt/gentoo/boot

The following mount commands must be run to make it possible to run portage to update the system’s software. I have fixed issues without these mounts, but I have also received some cryptic errors. Remember that you may need to create directories if they do not exist.

livecd ~ # mount --types proc /proc /mnt/gentoo/proc
livecd ~ # mount --rbind /sys /mnt/gentoo/sys
livecd ~ # mount --make-rslave /mnt/gentoo/sys
livecd ~ # mount --rbind /dev /mnt/gentoo/dev
livecd ~ # mount --make-rslave /mnt/gentoo/dev

Next, we need to copy the DNS information to the chroot environment to ensure that we will have a working network. You can skip this step if you think that the installed system connected to the internet just fine. Additional configuring of the network may be required if it doesn’t come up on its own. This page of the installation guide can help. You may now see how the arch-chroot script really does simplify the process.

livecd ~ #cp --dereference /etc/resolv.conf /mnt/gentoo/etc/

The next four commands will do the actual chrooting and set up the environment.

livecd ~ # cd /mnt/gentoo
livecd /mnt/gentoo # chroot /mnt/gentoo /bin/bash
livecd / # source /etc/profile
livecd / # export PS1="(chroot) ${PS1}"
(chroot) livecd / #

We have now chrooted into the installed Gentoo system. Notice that the last command has changed the system prompt. You could now run portage commands (emerge) or do whatever you wish to the system. Be careful you are logged in as root.

To exit this chroot enter the command to Exit followed by the umount commands to unmount all of the partitions and devices which you mounted to enter chroot.

(chroot) livecd / # exit
livecd /mnt/gentoo # cd
livecd ~ # umount -l /mnt/gentoo/dev
livecd ~ # umount -l /mnt/gentoo/sys
livecd ~ # umount -l /mnt/gentoo/proc
livecd ~ #umount -l /mnt/gentoo/boot
livecd ~ #umount -R /mnt/gentoo
livecd ~ # 

Finally remove the image and reboot…or use shutdown -h now to perform a full system shutdown.

livecd ~ # reboot

Finally…

If your installed system uses systemd some of the tools such as hostnamectl or timedatectl will not work. You are basically running on whatever processes the image used to boot up. Chroot is used best as a rescue tool and not to fully boot up a damaged system for normal usage.