Recovering and Configuring a ThinkLogical SCS Console Server

May 23
6 min read

What this is

A ThinkLogical Secure Console Server (SCS) is a rack-mounted serial aggregator. You plug servers, routers, switches, and other devices into its serial ports, and then access those devices' console ports remotely over SSH. Think of it as a central hub for out-of-band management. The SCS320 model supports 32 serial ports. Thinklogical made many different models with various serial ports available, including modular power supplies and a slew of very interesting other products outside of console servers.

These units run a stripped-down Red Hat Linux 8.0 off a 512 MB CompactFlash (CF) card. They're old, they're off support, and they occasionally show up used without credentials. This document covers how to recover one from scratch, get it configured, and understand why the obvious recovery approach doesn't work.

TL;DR >>> how to actually recover it

The system mounts a tmpfs over /etc at boot and populates it from /misc/etc.tar.gz. All your edits to /etc/* on the CF card are silently ignored at runtime. You have to modify the tarball itself.

1. Pull the CF card and connect it to a Linux machine via a USB CF adapter. Run lsblk to identify the device. Look for a ~512MB disk, usually /dev/sda or /dev/sdb. It will have three partitions. You want partition 2 (the root filesystem).

Example lsblk output:

sda  488.7M
├─sda1 9.8M   /boot
├─sda2   464.1M   /<-- this one
└─sda314.8M   /recover

sudo mount /dev/sda2 /mnt/cf

(replace sda2 with your actual partition)

2. Back up the tarball before touching anything.

cp /mnt/cf/misc/etc.tar.gz /mnt/cf/misc/etc.tar.gz.bak

3. Extract the tarball into a working directory.

mkdir /tmp/etc_work
sudo tar -zxpf /mnt/cf/misc/etc.tar.gz -C /tmp/etc_work --same-owner

The --same-owner flag preserves the original uid:gid on files inside the archive. Do not skip this or you will break authentication.

4. Generate an MD5-crypt password hash. RHL 8.0 era PAM expects the $1$... format. Modern bcrypt won't work here.

openssl passwd -1 -salt "recovery" "your_new_password"

Output looks like: $1$recovery$XXXXXXXXXXXXXXXXXXXXX/

5. Set the root password in the extracted shadow file. Replace the hash below with the actual output from step 4.

sudo sed -i 's|^root:[^:]*:|root:$1$recovery$XXXXXXXXXXXXXXXXXXXXX/:|' /tmp/etc_work/shadow

6. Add your own user account. UID 500 is taken by the built-in 'conserver' account. Use 501 or higher.

Add a line to passwd:

echo 'yourusername:x:501:501::/home/yourusername:/bin/bash' >> /tmp/etc_work/passwd

Add the shadow entry (use the same hash generation method as step 4):

echo 'yourusername:$1$recovery$XXXXXXXXXXXXXXXXXXXXX/:19870:0:99999:7:::' >> /tmp/etc_work/shadow

Add the group entry:

echo 'yourusername:x:501:' >> /tmp/etc_work/group

Add to wheel group so sudo works (find the wheel line and add your username):

sudo sed -i 's/^wheel:x:10:/wheel:x:10:yourusername/' /tmp/etc_work/group

Create a home directory inside the tarball:

sudo mkdir -p /tmp/etc_work/home/yourusername
sudo chown 501:501 /tmp/etc_work/home/yourusername

7. Repack the tarball, preserving ownership. Run this as root. Do NOT pass --owner=root or you will destroy file ownership.

cd /tmp/etc_work
sudo tar -czpf /mnt/cf/misc/etc.tar.gz .

8. Clean unmount and reinsert.

sync && sudo umount /mnt/cf

Put the CF card back in the SCS, power it on, and wait for it to boot. The front panel LCD will show "expand /etc" while it extracts the tarball. That's normal. Once it shows the IP address, you're ready to connect.

Connecting via SSH

The SCS runs an old SSH daemon that uses deprecated key exchange algorithms. Modern SSH clients will reject it by default. Use the following flags:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 \
-oPubkeyAcceptedKeyTypes=+ssh-rsa \
-oHostKeyAlgorithms=+ssh-rsa \
yourusername@<ip-address>

To avoid typing this every time, add an entry to ~/.ssh/config on your workstation:

Host <ip-address>
	KexAlgorithms +diffie-hellman-group1-sha1
	PubkeyAcceptedKeyTypes +ssh-rsa
	HostKeyAlgorithms +ssh-rsa

After that, a plain ssh yourusername@<ip-address> will work.

First login cleanup

When you first log in you may see:

unknown terminal "xterm-256color"

This is harmless. The 2002-era terminfo database doesn't know about modern terminal types. Fix it by adding export TERM=xterm to your shell profile (see below).

You may also see a home directory permission error if the user was added manually to the tarball rather than through adduser. Fix it as root:

chmod 755 /home/yourusername
chown yourusername:yourusername /home/yourusername

Getting root access

Your user account needs to be in the wheel group for sudo to work. As root:

usermod -G wheel yourusername

Verify sudoers has the wheel group uncommented:

visudo

Look for this line and make sure it is not commented out:

%wheel  ALL=(ALL)   ALL

Initial configuration

Set the IP address

Use the front panel buttons and LCD display to set the IP address, netmask, and gateway. The front panel is self-prompting. Press the leftmost button to enter edit mode.

Fix DNS

The unit may have stale DNS servers from a previous owner. Update /etc/resolv.conf:

cat > /etc/resolv.conf << 'EOF'
nameserver x.x.x.x
nameserver x.x.x.x
EOF

Set up NTP

The default NTP config points to servers that no longer exist. Fix both the main config and the step-tickers file:

echo "server pool.ntp.org" >> /etc/ntp.conf
echo "pool.ntp.org" > /etc/ntp/step-tickers
service ntpd start

NTP is already configured to start on boot (runlevels 3, 4, and 5), so no further action is needed there.

Verify the time is correct:

date

Persisting your changes

This is critical and easy to forget. Because /etc is a tmpfs at runtime, any changes you make will be lost on reboot unless you explicitly save them. The SCS has a built-in save command that handles this. When you're root, it should be in your path, but if it's not found, you should be able to find it in /sbin/save.

save

Run this as root after any configuration change you want to keep. It tars up the current /etc state, rotates the old backups in /misc/, and writes the new tarball to /misc/etc.tar.gz. The previous four versions are kept as etc.tar.gz.1 through etc.tar.gz.4.

If you forget to run save before rebooting, your changes are gone.

Normal operation

The SCS command set is documented in man lsi. The main commands you'll use day to day:

lsp : List all configured device port names

lsc : Show currently active connections

connect port1 : Attach interactively to a device port

monitor port1 : Watch traffic on a port without sending

stty : View or change serial port settings

To connect to a device: SSH into the SCS, run lsp to see port names, then connect <portname>. You'll have an interactive console session to whatever is plugged into that port.

To disconnect from a port and return to the SCS shell: press ESC then Shift+A (the escape sequence is \x1bA).

Making a full backup of the CF card

Before making any significant changes, image the entire CF card. With the card mounted on a Linux machine:

# lsblk will show the CF card as a whole disk (e.g. sda), not a partition.
# Use the disk device, not a partition number.
sudo dd if=/dev/sda of=~/scs320-backup.img bs=4M status=progress

This creates a complete byte-for-byte image. To restore, reverse the if and of arguments. Store the image somewhere safe.

Why shadow edits don't work; the full explanation

The boot architecture

The CF card has three ext2 partitions:

/dev/sda1  →  /boot  (kernel and GRUB bootloader)
/dev/sda2  →  /  (root filesystem, RHL 8.0 userspace)
/dev/sda3  →  /recover   (empty recovery partition)

At the root of the filesystem there are some unusual symlinks:

/home  ->  /etc/home
/root  ->  /etc/root
/var   ->  /etc/var

This is the first clue something unusual is happening. On a normal Linux system these would be real directories.

The tmpfs overlay

The answer is in /etc/rc.d/rc.sysinit. During boot, around line 244:

if [ $COMPACTFLASH -eq 1 ]; then
	mount -n tmpfs /dev/shm -t tmpfs -o size=32m
	mkdir /dev/shm/etc
	tar -zxf /misc/etc.tar.gz -C /dev/shm/etc
	mount -n --bind /dev/shm/etc /etc
fi

What this does:

1. Creates a RAM-based filesystem (tmpfs) in /dev/shm

2. Extracts /misc/etc.tar.gz into that RAM filesystem

3. Bind-mounts that RAM filesystem over /etc

The result: the /etc the running system sees is entirely in RAM, populated from the tarball. The actual /etc directory sitting on the CF card is completely bypassed. Any edits you made directly to files on the CF are invisible to the running system.

This also explains the symlinks. Since /etc is the only writable location at runtime (everything else on the CF is read-only), /home, /root, and /var all point into subdirectories of /etc so they can be written to.

It also explains why IP address changes made through the front panel persist across reboots: the front panel LCD daemon writes changes back into /misc/etc.tar.gz directly. The tarball is the persistence layer for the entire system.

Why the shadow file on the CF had no effect

When you pull the CF, edit /etc/shadow directly on the card, and put it back, you're editing the on-disk copy that the boot process completely ignores. The running system loads shadow from the tarball instead. The same applies to /etc/passwd, /etc/ssh/sshd_config, PAM config, and everything else under /etc.

The tarball is the only thing that matters.

Lessons

1. Trace the boot sequence when edits don't take. Read rc.sysinit or its modern equivalent. Anything that does mount --bind or extracts a tarball into a runtime location is silently overriding whatever is on disk. This pattern is common in embedded appliances.

2. The symlinks were the canary. /var -> /etc/var should immediately raise the question "where do logs go?" and the answer leads straight to the tmpfs overlay.

3. PAM config files existing is not evidence they are active. The unit ships with LDAP config files present but LDAP is not wired into nsswitch.conf or pam.d/system-auth. Authentication falls through to local shadow via pam_unix.so. The LDAP database directory is empty.

4. The front panel narrates the boot. The LCD showing "expand /etc" during boot is the system telling you exactly what it is doing. That message, combined with reading the script that prints it, is what makes everything click.