Pebbles in the Sand

Linking shared libraries: overcoming runtime errors

kawakamasu — Wed, 22 Apr 2009 16:37:44 +0000

Say you’re developing a program in C++ and you need to use some exotic library called libAAA.
You have been able to download the sources and build the package ( by adding -I/path/to/the/library/include to the compilation options, for example) and linking it ( by adding something like -L/path/to/the/library/lib -lAAA to the linker options, for example).

But running the program raises an error:>>./program_a ./program_a: error while loading shared libraries: libAAA.so: cannot open shared object file: No such file or directory

This is due to the fact that although you have informed the compiler and the linker about the location of your libAAA files, the system cannot find them at runtime. In fact, the program_a would call a dynamic linker that searches for shared libraries in the system. It, however, searches only in default locations where libraries are commonly installed (such as /usr/lib).


You can check the dynamic libraries by invoking ldd command on the executable:

>>ldd program_a

        linux-gate.so.1 =>  (0xb7f48000)

        liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0xb7836000)

        libAAA.so => not found

        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb76c4000)

        /lib/ld-linux.so.2 (0xb7f49000)



Here it is clearly seen that the linker cannot find the library. A remedy is to add the custom library path to the LD_LIBRARY_PATH environment variable (this example is for bash shell):

>>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/the/library/lib



 Now the library can be located at runtime:

>>ldd program_a

        linux-gate.so.1 =>  (0xb7f48000)

        liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0xb7836000)

        libAAA.so => /path/to/the/library/lib/libAAA.so (0xb77da000)

        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb76c4000)

        /lib/ld-linux.so.2 (0xb7f49000)

 
Another option is to do a manual system-wide install by copying the library to /usr/lib, for example, or modifying the dynamic linker look-up path (in file /etc/ld.so.conf). Naturally such problems should not appear if the library is well-built so that make install will do a system-wide installation automatically.
More information can be found in here for instance.



Duplex printing for HP Color printers in Ubuntu
kawakamasu — Thu, 16 Apr 2009 10:38:09 +0000

In office we have this fast HP Color LaserJet CP3505 printer, that should be able to print on both sides. I installed the driver using some GUI (system-config-printer I think) and everything worked like a charm, except the duplex printing. Trying to enable duplex printing in kprinter, for example, resulted an error “Some options selected are in conflict” as the Duplex Unit was not installed.
The printer setup can be changed (as instructed in this thread) in the CUPS HTML interface, accessible via any browser:
http://localhost:631
1) Choose Manage Printers and Set Printer Options of the printer at hand.
2) In Options Installed change radio button to Duplex Unit : Installed.
3) Click Set Printer Options.

You may be asked for a user name and password. Type in your local login user name and password. Sometimes, as in my case, you may need to input root and the superuser password.
Presumably the same configuration can be done in the GUIs as well, for example:

sudo system-config-printer and tick Duplex Unit in Installable Options tab. (In KDE use kdesu instead of sudo.)
       


Floppy drive access in Ubuntu Hardy Heron
kawakamasu — Wed, 24 Sep 2008 12:32:48 +0000

For those who still need to use floppies, it seems that floppy drives are not fully supported in Ubuntu anymore as there is no out-of-the-box solution that would work.
However, you can always mount the floppy manually:

1) Make sure that the floppy drive is properly connected to motherboard and that (internal) floppy access is enabled in system BIOS.

2) Start Ubuntu and create directory /media/floppy or something similar.

3) Mount the floppy by sudo mount /dev/fd0 -t auto /media/floppy.
You should be able to access the floppy either trough terminal or GUI programs.

Make sure to unmount manually when you’re done to prevent data loss: sudo umount /media/floppy.
Note that the GUI access *does not* work so do not try to access the floppy via Nautilus or Konqueror or such before mounting manually (step 2).
       


Recovering files from ext3 filesystem with foremost
kawakamasu — Fri, 15 Aug 2008 13:59:10 +0000

So you accidentally removed some files on Linux ext3 partition and you want them back.

Here’s a procedure that worked for me:
1) Get foremost recovery tool. In Ubuntu it is in the universe repositories.

2) Unmount the partition where the files were. If it’s the root partition (say, /dev/sda1) , you cannot unmount it. In this case use Ubuntu Live CD or such to reboot into a system that does not mount the /dev/sda1.

3) Run (note that you must run with sudo):

sudo foremost -i /dev/sda3.
foremost will create a directory called output where all the recovered files are stored, separated in subdirectories according to filetype. If you need to recover only certain files, use the -t option:

sudo foremost -t jpeg,gif -i /dev/sda3

would only recover jpeg and gif files.
       


kdm cannot see logins provided by NIS (Hardy Heron)
kawakamasu — Fri, 04 Jul 2008 11:15:36 +0000

At boot-up, KDE login manager (kdm) shows only local logins, but none of those that are provided by a remote Network Information Service (NIS, it’s yellow page server, in fact). One needs to manually restart kdm:
sudo /etc/init.d/kdm restart,
which gets frustrating in the long run. It seems that for some reason, kdm is loaded before the NIS services and thus it cannot see the remote logins. To fix this, move kdm later in the startup process:


cd /etc/rc2.d

sudo mv S13kdm S30kdm


This workaround worked for me. Postponing the kdm startup seems to have no side effects. 
       


Japanese input method in KDE 3.5 (Hardy Heron)
kawakamasu — Tue, 27 May 2008 23:32:35 +0000

The easiest (and recommended) way to enable Japanese input in KDE is through System Settings -> Regional & Accessibility -> Country/Region & Language. Installing support for Japanese also installs the scim-anthy input method. If you need to use Japanese in terminal, the easiest way is to set Japanese as system language (but that indeed changes the default language, e.g. most man pages and system commands like apt-get will be in Japanese).
However, this will not enable Japanese input in non-KDE programs like Firefox. For this you need to install scim-bridge-client-gtk and im-switch. Then run
im-switch -s scim-bridge
to set scim-bridge as default input method for all applications. 
scim-bridge is a new socket based input method module that fixes the many annoying problems that were present with scim, like various crashes in Firefox and Thunderbird and whitespace mapping bugs in KDE. The only thing that doesn’t work is Japanese input in Skype, which is a pity. Apparently this is fixed in KDE 4.0.
All in all this is very painless procedure compared to what it used to be in Dapper or Edgy.
       


A srcipt for running processes in parallel in Bash
kawakamasu — Thu, 22 May 2008 17:40:12 +0000

In Bash you can start new processes (theads) on the background simply by running a command with ampersand &. The wait command can be used to wait until all background processes have finished (to wait for a certain process do wait PID where PID is a process ID). So here’s a simple pseudocode for parallel processing:
for ARG in  $*; do
    command $ARG &
    NPROC=$(($NPROC+1))
    if [ "$NPROC" -ge 4 ]; then
        wait
        NPROC=0
    fi
done

I.e. you run 4 processes at a time and wait until all of them have finished before executing the next four. This is a sufficient solution if all of the processes take equally long to finish. However this is suboptimal if running time of the processes vary a lot.
A better solution is to track the process IDs and poll if all of them are still running. In Bash $! returns the ID of last initiated background process. If a process is running, the corresponding PID is found in directory /proc/.
Based on the ideas given in a Ubuntu forum  thread and a template on command line parsing, I wrote a simple script “parallel” that allows you to run virtually any simple command concurrently.
Assume that you have a program proc and you want to run something  like proc *.jpg using three concurrent processes. Then simply do



parallel -j 3 proc *.jpg



The script takes care of dividing the task. Obviously -j 3 stands for three simultaneous jobs.

If you need command line options, use quotes to separate the command from the variable arguments, e.g.



parallel -j 3 "proc -r -A=40" *.jpg



Furthermore, -r allows even more sophisticated commands by replacing asterisks in the command string by the argument:



parallel -j 6 -r "convert -scale 50%  * small/small_*" *.jpg



I.e. this executes convert -scale 50%  file1.jpg small/small_file1.jpg for all the jpg files. This is a real-life example for scaling down images by 50% (requires imagemagick).
Finally, here’s the script. It can be easily manipulated to handle different jobs, too. Just write your command between #DEFINE COMMAND and #DEFINE COMMAND END.
#!/bin/bash
NUM=0
QUEUE=""
MAX_NPROC=2 # default
REPLACE_CMD=0 # no replacement by default
USAGE="A simple wrapper for running processes in parallel.
Usage: `basename $0` [-h] [-r] [-j nb_jobs] command arg_list
 	-h		Shows this help
	-r		Replace asterix * in the command string with argument
	-j nb_jobs 	Set number of simultanious jobs [2]
 Examples:
 	`basename $0` somecommand arg1 arg2 arg3
 	`basename $0` -j 3 \"somecommand -r -p\" arg1 arg2 arg3
 	`basename $0` -j 6 -r \"convert -scale 50% * small/small_*\" *.jpg"

function queue {
	QUEUE="$QUEUE $1"
	NUM=$(($NUM+1))
}

function regeneratequeue {
	OLDREQUEUE=$QUEUE
	QUEUE=""
	NUM=0
	for PID in $OLDREQUEUE
	do
		if [ -d /proc/$PID  ] ; then
			QUEUE="$QUEUE $PID"
			NUM=$(($NUM+1))
		fi
	done
}

function checkqueue {
	OLDCHQUEUE=$QUEUE
	for PID in $OLDCHQUEUE
	do
		if [ ! -d /proc/$PID ] ; then
			regeneratequeue # at least one PID has finished
			break
		fi
	done
}

# parse command line
if [ $# -eq 0 ]; then #  must be at least one arg
	echo "$USAGE" >&2
	exit 1
fi

while getopts j:rh OPT; do # "j:" waits for an argument "h" doesnt
    case $OPT in
	h)	echo "$USAGE"
		exit 0 ;;
	j)	MAX_NPROC=$OPTARG ;;
	r)	REPLACE_CMD=1 ;;
	\?)	# getopts issues an error message
		echo "$USAGE" >&2
		exit 1 ;;
    esac
done

# Main program
echo Using $MAX_NPROC parallel threads
shift `expr $OPTIND - 1` # shift input args, ignore processed args
COMMAND=$1
shift

for INS in $* # for the rest of the arguments
do
	# DEFINE COMMAND
	if [ $REPLACE_CMD -eq 1 ]; then
		CMD=${COMMAND//"*"/$INS}
	else
		CMD="$COMMAND $INS" #append args
	fi
	echo "Running $CMD" 

	$CMD &
	# DEFINE COMMAND END

	PID=$!
	queue $PID

	while [ $NUM -ge $MAX_NPROC ]; do
		checkqueue
		sleep 0.4
	done
done
wait # wait for all processes to finish before exit

       


Print directory tree disk usage on the command line
kawakamasu — Tue, 20 May 2008 11:27:50 +0000

Suppose you have a large tree of directories containing lots of data (such as source code of a big project or numerical output of your simulations) and you need to estimate the total size of the whole tree. In graphical user interface this can be done by examining the directory properties. But as usual things can be done faster on the command line, where the suitable command is du (for Disk Usage).
An example (assuming that the root of your directory tree is called data)
du -h --max-depth=1 data

1.1G    data/soln

3.0M    data/binary

1.1G    data


The -h option tells du to use human readable format, i.e. MB, GB etc instead of bytes. Option --max-depth=1 means that only the first subdirectories are listed. For more info on the options run man du.

It is convenient to create an alias for shortening the long command such as

alias disku='du -h --max-depth=1'

For the alias to be present in all future sessions, add the line to your shell initialization file (for bash shell ~/.bashrc for example).
       


Keep processes running even if you close terminal / log out
kawakamasu — Sat, 17 May 2008 19:26:19 +0000

You left the office in a hurry on Friday and forgot to run one process. Of course you can ssh to your desktop from home and put it running. But the problem is that the process takes several hours (or days) to complete. Do you need to keep you terminal and ssh connection open for all that time??
The answer is no. There are several ways of detaching a process from any display and thus killing the connection will not kill the process. The easiest method that I have encountered is  screen. It is usually in the linux repositories, so for example in Ubuntu you can download it though apt. Note that you need to install it on the machine where want to run the executable.

Here's how it works.

1) Log in to the remote machine.

2) Run screen in command line. A new "virtual" shell will open (after you have pressed space or enter). The new shell is the one that will be detached.

3) Run your process.

4) Detach the display by pressing ctrl+a and ctrl+d in sequence. The original shell (where you typed screen) is displayed. You can now freely log out without affecting the detached process.

5) To resume to the virtual shell, log in again and run screen -r on the remote machine.

The detached shell is displayed again just as you left it. As if you had never logged out at all!

6) When you're finally done kill the virtual shell by running exit as usual.
It is also possible to use command nohup for this task, but it's not as advanced as screen and it stores all output to logfiles.
       



Using kompare to view Subversion differences
kawakamasu — Fri, 16 May 2008 16:00:15 +0000

When you’re using Subversion (svn) repositories for code development, it is sometimes useful to check differences between code revisions (i.e. when things went wrong for the first time). If you are using KDE, Kompare is a graphical difference viewer that can be easily used for this task. Simply run



svn diff -r 1020:1047 | kompare -o -



in a directory that belongs to the svn tree. Kompare window will open showing comparison of all the changed files in that directory tree. The numbers refer to revision numbers, i.e. in this case revisions 1020 and 1047 are being compared. The revision switch -r accepts other formulations, too:



'{' DATE '}' revision at start of the date

'HEAD'        latest in repository

'BASE'       base rev of item's working copy

'COMMITTED'  last commit at or before BASE

'PREV'       revision just before COMMITTED


For more information on the svn difference command run svn -h diff.