System Administration Tricks

setuid/setgid Binaries

(Linux Hacks Vol. I, #11)

Here's a find incantation for finding all the setuid/setgid binaries:

# find / -perm +6000 -type f -exec ls -ld {} \;

Manipulating Processes Symbolically

(Linux Hacks Vol. I, #17)

To freeze the user on terminal pts/2:

# skill -STOP pts/2

To release them from the grip of sleeping death, try this:

# skill -CONT pts/2

Or, to renuce all of luser's processes to 5:

# snice +5 luser

Cleaning up after Ex-Users

(Linux Hacks Vol. I, #19)

Lock the account:

# password -l luser

Change the users login shell (via SSH with keys)

# chsh -s /sbin/nologin luser

Note that this doesn't affect their ability to forward ports via SSH.

Generate a list of all owned files:

# find / -user luser -exec ls -ld {} \; > /root/users/luser/files.lst

Generate a list of all owned processes:

# ps -ef | grep -i ^luser > /root/users/luser/procs.lst

Rename the home directory to disable authorized keys, .rhosts, public_html, etc.:

# mv /home/luser /home/luser.suspended

Check cron and at jobs via crontab -lu and atq

Archiving with Pax

(Linux Hacks Vol. I, #39)

Create an archive (a la tar -cf):

$ pax -wvf backup.pax .
$ pax -f backup.pax | less
$ cd restore/ ; pax -rvf ../backup.pax

Interactive looks neat:

$ pax -rif ~/backup.pax

Also, recursive directory copy is cool:

$ pax -rq src/ dst/

Incremental backups are possible.

First, do a full backup on Monday:

$ cd ; pax -wvf /srv/backups/mon .

Then, on Tuesday, backup all the files since Monday:

$ cd ; pax -wv -T 0000 -f /srv/backups/tue .

Skip a file on restore:

$ pax -rvf ~/archive.pax -c './skipme'

Skip a set of files on restore:

$ pax -rvf ~/archive.pax -c '*skip'

Only restore a subset of files:

$ pax -rvf ~/archive.pax -n '*.good'

Automated Snapshot-style Incremental Backups with rsync

(Linux Hacks Vol. I, #42)

Here is a method for generating automatic rotating snapshot-style backups on a Linux server. Snapshot backups are a feature of some high-end industrial strength file servers; they create the illusion of multiple full (up to ownership/permission) backups per day without the space or processing overhead. All of the snapshots are read-only and are accessible directly by users as special system directories.

Since making a full copy of a large filesystem can be a time-consuming and expensive process, it is common to make full backups only one a week or once a month, and store only changes on the other days. This technique is called making "incremental" backups, and us supported by the venerable old dump and tar utilities along with many others.

The standard GNU fileutils cp command ocmes with a -l flag that causes it to create (hard) links instead of copies (it doesn't hard-link directories, though, which is good; you might want to think about why that is). Another handy switch for the cp command is -a (archive), which causes it to recurse through directories and preserve owners, timestamps, and access permissions.

Together, the combination cp -al makes what appears to be a full copy of a directory tree but is really just an illusion that takes almost no space. If we restrict operations on the copy to adding or removing (unlinking) files — i.e., never changing one in place — then the illusion of a full copy is complete. To the end-user, the only differences are that an illusion-copy takes almost no disk space and almost no time to generate.

We can combine rsync and cp -al to create what appear to be multiple full backups of a filesystem without taking multiple disks' worth of space, as in:

rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete src/ backup.0/

If the above commands are run once every day, then backup.0, backup.1, backup.2 and backup.3 will appear to each be a full backup of src/ as it appear today, yesterday, two days ago, and three days ago, respectively — complete, except that permissions and ownerships in old snapshots will get their most recent values (thanks to J.W. Schultz for pointing this out). in reality, the extra storage will be equal to the current size of src/ plus the total size of changes over the last three days — exactly the same space that a full plus daily incremental backup with dump or tar would have taken.

This method is much better for network-based backups, since it's only necessary to do a full backup once, instead of once per week. Thereafter, only the changes need to be copied. Unfortunately, you can't rsync to a tape; you'll still need dump or tar for that.

If you have a spare machine, even a very low-end one, you can turn it into a dedicated backup server. Make it standalone, and keep it in a physically separate place — another room or even another building. Disable every single remote service on the backup server, and connect it only to a dedicated network interface on the source machine.

You can then perform backups form this machine using rsync over ssh, and export the backups back to the original machine via read-only NFS. Then users can get to the snapshots themselves (without needing sysadmin intervention) and can't possibly change or delete them.

I'd consider this "pretty good" protection, but if you're (wisely) paranoid, or your job is on the line, build two backup servers. Then you can make sure that at least one of them is always offline.

Network Monitoring with ngrep

(Linux Hacks Vol. I, #60)

ngrep is grep for network packets:

# ngrep -d wlan0 -qi google.com

SSH Port-Forwarding

(Linux Hacks Vol. I, #71)

To forward ports:

$ ssh -f -N