glibc除了提供最底层的C运行库，还封装了kernel提供的API，程序通过glibc进行系统调用（ syscall）。
应用层面的C库，比如OpenSSL库： /usr/include/openssl比如函数SSL_set_session
查看手册 man SSL_set_session引入头文件 #include <openssl/ssl.h>标准C库，比如stdio库：
/usr/include/stdio.h 比如函数fprintf： 查看手册 man fprintf引入头文件 #include <stdio.h> Linux系统调用，比如epoll：/usr/include/x86_64-linux-gnu/sys/ epoll.h比如函数epoll_wait查看手册 man epoll_wait引入头文件 #include <sys/epoll.h>使用strace更直观地查看程序进程进行的系统调用：比如追踪Nginx的工作进程（假设其PID为1185）响应以下请求进行的系统调用：
curl -I http://127.0.0.1/生成报告：sudo strace -Tcp 1185| 1 2 3 4 5 6 7 8 9 10 11 12 | % time seconds usecs /call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- -nan 0.000000 0 1 write -nan 0.000000 0 1 close -nan 0.000000 0 1 writev -nan 0.000000 0 3 1 recvfrom -nan 0.000000 0 1 shutdown -nan 0.000000 0 3 epoll_wait -nan 0.000000 0 1 epoll_ctl -nan 0.000000 0 1 accept4 ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000000 12 1 total |
实时检测：sudo strace -Tp 1185
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | epoll_wait(13, { {EPOLLIN, {u32=18104944, u64=18104944}}}, 512, 4294967295) = 1 <3.288133> accept4(10, {sa_family=AF_INET, sin_port=htons(56261), sin_addr=inet_addr( "127.0.0.1" )}, [16], SOCK_NONBLOCK) = 3 <0.000037> epoll_ctl(13, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLET|0x2000, {u32=18105545, u64=18105545}}) = 0 <0.000024> epoll_wait(13, { {EPOLLIN, {u32=18105545, u64=18105545}}}, 512, 60000) = 1 <0.000057> recvfrom(3, "HEAD / HTTP/1.1\r\nUser-Agent: cur" ..., 1024, 0, NULL, NULL) = 162 <0.000057> writev(3, [{ "HTTP/1.1 400 Bad Request\r\nServer" ..., 157}], 1) = 157 <0.000106> shutdown (3, 1 /* send */) = 0 <0.000082> recvfrom(3, 0x7fff07f92f10, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) <0.000023> epoll_wait(13, { {EPOLLIN|EPOLLHUP|0x2000, {u32=18105545, u64=18105545}}}, 512, 5000) = 1 <0.000039> recvfrom(3, "" , 4096, 0, NULL, NULL) = 0 <0.000021> write(8, "127.0.0.1 - - [14/Feb/2014:17:24" ..., 173) = 173 <0.000055> close(3) = 0 <0.000030> epoll_wait(13, |
注意：跟踪会使进程的运行效率明显下降。