Error while loading shared libraries
As a Linux C/C++ developer did you ever encounter the error while loading shared libraries error when launching an executable, even though the apparently missing library is located in the same folder as the executable? In this blog we investigate WHY that happens, and how to solve it in an alternative way than the “rpath option of the linker”-method.
When this problem pops-up the most common solution is to use the rpath option of the linker in the setting of the LD_LIBRARY_PATH. There are some examples here and here.
In this post we will try to understand why this happens and look at a different way we can solve the issue.
Aside: Jupyter Notebook
This post was originally written using Jupyter Notebook (if you haven’t heard about it already, you should check it out, its awesome!).
Below you will see some strange-looking annotations, such as %%file these are called magic commands, here is an overview of the ones we use in this post:
%%file <filename>: Will write a file called<filename>to the file system.! <command>: Will run<command>in the default shell.var = ! <command>: Will run<command>in the default shell and capture the output in a variable calledvar. This is useful for post-processing the output from the shell command.
Get and overview of using magic commands in Jupyter Notebook here. Note: in the link the author refers to IPython, this was the previous name for the Jupyter project.
The great thing about these magic commands is that you execute the commands, so you can actually create a file on your computer and run it using these commands. This give you the opportunity to tag along as you read this blog post. Unfortunately this does not work in this particular blog post as it is written in Markdown but you can visit the original blog post here, where it works: [Coming soon!]
Reproducing the problem: Our test setup
Lets create a small library that we can experiment with, we will call it amazum.
We start with the header:
1
2
3
4
5
%%file amazum.hpp
#pragma once
int sum(int a, int b);
Then we create a C++ file where we include the header:
1
2
3
4
5
6
7
8
%%file amazum.cpp
#include "amazum.hpp"
int sum(int a, int b)
{
return a + b;
}
First step in creating a shared library (.so file on Linux) is to compile the code into an object file, in this case amazum.o. To do this we pass the following options to g++ (compiler for C++ code available on most Linux systems):
1
! g++ -fPIC -c amazum.cpp
-fPICIs used to create Position Independent Code (PIC), this should be used for shared libraries as these can be loaded in memory at arbitrary locations.-cCreate an object file.amazum.cppThe input file we want to compile.
This will produce the amazum.o object file from which we can instruct g++ to create the shared library, by passing the following options:
1
! g++ -shared -fPIC -o libamazum.so amazum.o
-sharedProduce a shared library.-fPICThis option must be repeated if specified in the compilation step.-o libamazum.soThis option allows us to specify the output file name.amazum.oThe input file.
To reproduce the problem we need to create a small program that links against our newly created library.
1
2
3
4
5
6
7
8
9
10
%%file program.cpp
#include "amazum.hpp"
#include <iostream>
int main()
{
std::cout << "The sum of 324 + 423 is = " << sum(324, 423) << std::endl;
return 0;
}
We now have a small program that uses the amazum library to compute the sum of two numbers and print the result. Next stop is to compile and link our program, this is again done using g++ by passing the following options:
1
g++ -o program program.cpp libamazum.so
-o programSets the output file name.program.cpp libamazum.soInput files to the compilation.
Alternatively you can write the following:
1
! g++ -Wall -o program program.cpp -L. -lamazum
The two commands do exactly the same thing, but in the latter we explicitly specify the library path using -L and libraries -l
Lets try to run the program and see what happens:
1
! ./program
./program: error while loading shared libraries: libamazum.so: cannot open shared object file: No such file or directory
As expected, something went wrong, our library could not be found. The name was correct, but for some reason it was not found - lets investigate…
One initial question we may raise is: Where is it specified which shared libraries our program needs to load? Somehow this must be encoded into the file format of our binary, but what is the used file format?.
Lets start by answering the second question first. As it turns out this is quite easy to determine using the file utility on Linux (see file man page).
1
! file program
program: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9ca00f118f4ac50b6e9a518d3272845aef38033e, not stripped
So now we know that our executable is stored in the ELF file format.
The file utility shows a bunch of other pieces of information, but those are not important for us to be able to answer our initial question. You can read all about the ELF file format in the elf man page.
A quick Google search for where the shared libraries might hide in the ELF file lead us to this page (search for: Dynamic linking and the ELF interpreter). Turns out that the shared libraries used by a binary is stored in a section called the dynamic section. We can read this section by passing the -d option to the readelf utility, like so:
1
2
dynamic_section = ! readelf -d program
dynamic_section.grep('NEEDED')
[' 0x0000000000000001 (NEEDED) Shared library: [libamazum.so]',
' 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]',
' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]']
If you want to know more about the readelf utility you can read it here:(readelf man page).
In the above we use the grep command to only get the lines including NEEDED these are the required shared object dependencies for our program. As expected we see our shared library libamazon.so listed there.
Loading shared libraries is performed by a program called a dynamic linker. Lets visit the man page for ld.so, here we can read how the dynamic linker finds the shared libraries:
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time). If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
o (ELF only) Using the directories specified in the DT_RPATH dynamic
section attribute of the binary if present and DT_RUNPATH
attribute does not exist. Use of DT_RPATH is deprecated.
o Using the environment variable LD_LIBRARY_PATH (unless the
executable is being run in secure-execution mode; see below). in
which case it is ignored.
o (ELF only) Using the directories specified in the DT_RUNPATH
dynamic section attribute of the binary if present.
o From the cache file /etc/ld.so.cache, which contains a compiled
list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z
nodeflib linker option, shared objects in the default paths are
skipped. Shared objects installed in hardware capability
directories (see below) are preferred to other shared objects.
o In the default path /lib, and then /usr/lib. (On some 64-bit
architectures, the default paths for 64-bit shared objects are
/lib64, and then /usr/lib64.) If the binary was linked with the
-z nodeflib linker option, this step is skipped. The dynamic linker
Quite a bit of text, but by reading the above we get an overview of the various ways we can get the dynamic linker to find our library. The first options sounds interesting; if the shared object dependency contains slash the linker will interpret that as a relative/absolute path.
Lets explore that option, our hypothesis is that if we can make the dependency string contain a / then the dynamic linker should try to locate our library relative to our program.
Experiment one: Using ./amazum.so when building our program.
This is motivated by the sentence:
(this can occur if a shared object pathname containing slashes was specified at link time)
1
! g++ -g -Wall -o program program.cpp ./libamazum.so
Lets use readelf to check the result.
1
2
dynamic_section = ! readelf -d program
dynamic_section.grep('NEEDED')
[' 0x0000000000000001 (NEEDED) Shared library: [./libamazum.so]',
' 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]',
' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]']
Excellent!, notice that we now have a relative path in front of our library name. Let’s try to run our program.
1
! ./program
The sum of 324 + 423 is = 747
It worked! Lets see if we can do the same using the other approach of specifying the required libraries (using the -L and -l options).
1
! g++ -g -Wall -o program program.cpp -L./ -lamazum
Lets use readelf to check the result.
1
2
dynamic_section = ! readelf -d program
dynamic_section.grep('NEEDED')
[' 0x0000000000000001 (NEEDED) Shared library: [libamazum.so]',
' 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]',
' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]']
Not so good, no path ./ is in front for the libamazum.so library name.
Experiment two: Setting the SONAME of a shared object
From where does the binary get the name libamazum.so? And is it possible to modify this to include the ./? Lets review the linker man page to see how to set the SONAME, consult man 1 ld and search for -soname to find the following description:
-h name
-soname=name
When creating an ELF shared object, set the internal DT_SONAME
field to the specified name. When an executable is linked with a
shared object which has a DT_SONAME field, then when the
executable is run the dynamic linker will attempt to load the
shared object specified by the DT_SONAME field rather than
using the file name given to the linker.
So by passing either -h or the more descriptive -soname option to the linker we can set the internal SONAME of our libamazum.so library. Which should then be in the binary’s dynamic section.
Lets recompile the library passing the -soname option to the linker. This is a bit tricky to do, if you check the man page for g++ you will find that you can pass options to the linker by passing -Wl,[some-option].
1
! g++ -shared -o libamazum.so amazum.o -Wl,-soname=./libamazum.so
Now, relink our binary with the new shared library:
1
! g++ -g -Wall -o program program.cpp libamazum.so
Final step is check the shared object names in the dynamic section of our binary:
1
2
dynamic_section = ! readelf -d program
dynamic_section.grep('NEEDED')
[' 0x0000000000000001 (NEEDED) Shared library: [./libamazum.so]',
' 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]',
' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]']
Looks good! but can we run the program:
1
2
3
! ./program
The sum of 324 + 423 is = 747
Yes, it worked! Lets try with the alternative way of specifying the required libraries:
1
2
3
4
! g++ -g -Wall -o program program.cpp -L./ -lamazum
dynamic_section = ! readelf -d program
dynamic_section.grep('NEEDED')
[' 0x0000000000000001 (NEEDED) Shared library: [./libamazum.so]',
' 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]',
' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]']
Looks good, our library has the right ./ in front of its name. Let try to run it:
1
2
3
! ./program
The sum of 324 + 423 is = 747
Conclusion
In this post we have shown a different solution to loading shared libraries from a folder relative to the binary that avoids using the rpath and LD_LIBRARY_PATH approaches.
But we also hope you found our method of problem solving interesting and relevant for solving problems in the future.