Knowledgebase

Introduction

Like physical servers, virtual servers sometimes need rescue. System problems can occur for many reasons. The system may run out of space, Misbehaving applications can corrupt the disk, or failed updates may leave the system unbootable. If the system isn't accessible, use a bootable rescue ISO to resolve the issue.

Make a Backup

Before attempting any rescue, back up your VPS. RCS offers manual snapshots and automated backups for VPS instances. If you do not have automatic backups enabled, make a manual snapshot. Attempting to rescue a server without a backup is reckless. There is no substitute for a sound backup strategy. You can create a server snapshot in a few clicks in the customer portal.

You are responsible to back up all your mission-critical data, following your local laws and organizational policies. RCS does not maintain additional backups outside your customer portal.

Mount a Bootable ISO

There are several bootable ISOs in the RCS ISO library suitable for system rescue work.

• Finnix

• SystemRescue x64

You may choose any ISO in the RCS ISO library, or upload one. To attach an ISO to your server, go to the Server Information page in the customer portal. Select Settings > Custom ISO. Select your ISO, then click Attach ISO and Reboot.

Your server will reboot from the ISO. Connect to your server with the web console to perform the rescue steps.

Note: The Finnix ISO will time out and revert to the VPS operating system if you wait more than 60 seconds. If this happens, click the Send CtrlAltDel button at the top-right of the console to reboot the server to the ISO boot screen again. The SystemRescue x64 ISO does not revert back to the VPS operating system after booting.

Connect via SSH

The RCS web console is limited by the capabilities of noVNC, which is HTML-based. For easier access to the server, enable SSH in your rescue environment.

Finnix is lightweight and useful for low-memory environments, but many of its tools are outdated. For more about Finnix, see the RCS Doc, "Using Finnix Rescue CD to Rescue, Repair, or Backup Your Linux System".

If possible, we recommend using SystemRescue x64 for most tasks. The majority of examples in this guide use SystemRescue x64.

To enable SSH in SystemRescue x64:

1. Boot the SystemRescue x64 ISO.

2. Select Boot SystemRescue using default options

3. Set the root password with passwd.

   # passwd
   
   Enter new UNIX password:
   
   Retype new UNIX password:
   
   passwd: password updated successfully
   

Until the next reboot, SystemRescue x64 uses the new password. Setting the root password for SystemRescue x64 does not permanently change the root password of your server.

1. SystemRescue x64 automatically starts the SSH server. You can stop it with systemctl sshd stop or restart it with systemctl sshd restart.

Connect to the server as root with an SSH client, or use SFTP to access mounted disks. Locate the server IP address on the server information page. If you have previously connected to the server over SSH, be aware that the server's fingerprint will be different. You may need to remove the old entry from ~/.ssh/known_hosts before SSH connects.

Common Linux Rescue Scenarios

Locate the name of the VPS disk device

Use lsblk to locate your VPS disk and partition device names.

# lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

loop0    7:0    0 836.9M  1 loop /run/archiso/sfs/airootfs

sr0     11:0    1   888M  0 rom  /run/archiso/bootmnt

vda    254:0    0    55G  0 disk

└─vda1 254:1    0    55G  0 part

This example shows vda1 is a 55GB partition. Inspect further with blkid.

# blkid

/dev/vda1: UUID="10c173ba-bc37-4b35-a771-976098b870bc" TYPE="ext4" PARTUUID="bd17e170-01"

/dev/sr0: UUID="2019-02-09-12-53-31-00" LABEL="SYSRCD601" TYPE="iso9660" PTUUID="1e4ed97b" PTTYPE="dos"

/dev/loop0: TYPE="squashfs"

/dev/vda1 is the only ext4 partition on the server, confirming that /dev/vda1 is the VPS data device. Your server configuration may be different, make sure you clearly understand your data devices and names before proceeding.

Check the filesystem

Use fsck to check and correct filesystem corruption.

1. Use df -h to see the mounted disks.

   # df -h
   
   Filesystem      Size  Used Avail Use% Mounted on
   
   tmpfs           739M 1016K  738M   1% /media/ramdisk
   
   /dev/sdh        160M  160M     0 100% /media/sdh
   
   /dev/loop0      146M  146M     0 100% /media/compressed_root
   
   unionfs         739M 1016K  738M   1% /
   
   devtmpfs         10M     0   10M   0% /dev
   

2. Verify the VPS disk (vda in this example) is not in the list of mounted devices. Never run fsck on a mounted disk. Do not continue until you are sure the disk is unmounted.

3. Check the filesystem with fsck.

   # fsck -f /dev/vda1
   

4. If no problems are detected:

   # fsck -f /dev/vda1
   
   fsck from util-linux 2.33.1
   
   e2fsck 1.44.5 (15-Dec-2018)
   
   Pass 1: Checking inodes, blocks, and sizes
   
   Pass 2: Checking directory structure
   
   Pass 3: Checking directory connectivity
   
   Pass 4: Checking reference counts
   
   Pass 5: Checking group summary information
   
   /dev/vda1: 114923/3520000 files (0.1% non-contiguous), 907391/14417659 blocks
   

5. If errors are found, fsck will request permission to fix the issue.

   Superblock has an invalid journal (inode 4).
   
   Clear<y>?
   
   Press enter to automatically attempt to fix the problems.
   

6. Remove the ISO and reboot your server to verify the problem is resolved once fsck completes.

Mount VPS filesystem

Mount /dev/vda1 to a folder if you'd like to access the files directly. This mounts the vda1 partition to /mnt.

# mount /dev/vda1 /mnt

Browse /mnt to see the entire root filesystem. Use SCP or SFTP to transfer files from the system.

Change Root

Changing root (chroot) changes the apparent disk root directory, which is useful for system maintenance, such as reinstalling the bootloader or resetting passwords. This topic is complex. We recommend reviewing "What's the proper way to prepare chroot to recover a broken Linux installation?" to set up chroot properly. Once chroot is enabled, you can change user passwords, remove or install packages, and other tasks.

Check VPS system logs

Check the log files for clues about server problems. For example:

1. Verify the logfiles exist and have current dates.

   # ls -l /mnt/var/log
   

2. View the last 500 lines of syslog.

   # tail -n 500 /mnt/var/log/syslog | less
   

Check VPS disk space / filesystem inodes

If the VPS disk runs out of either blocks or inodes, your server will not function properly.

1. Check free blocks on device /dev/vda1 with df -h.

   # df -h /dev/vda1
   
   Filesystem      Size  Used Avail Use% Mounted on
   
   /dev/vda1        55G  2.6G   49G   5% /mnt
   

2. Check the inodes on device /dev/vda1 with the -i switch.

   # df -i /dev/vda1
   
   Filesystem      Inodes  IUsed   IFree IUse% Mounted on
   
   /dev/vda1      3520000 114923 3405077    4% /mnt
   

If the server is out of either blocks or inodes, delete some files or move them to another device.

Repair the Linux Bootloader

Bootloader problems vary by distribution, and specific instructions are beyond this short troubleshooting guide. This summary of GRUB and GRUB2 repair is useful if you are using SystemRescue x64. Here are some guides for popular distributions.

• Arch

• CentOS

• Debian

• Fedora

• Ubuntu

If your distribution isn't listed, we recommend a web search for "how to repair boot loader [distro name]".

Windows Password Reset

The RCS Doc "Reset Windows Server Administrator Password" explains how to reset a lost Windows administrator password.