Thread cancellation is a bit trickier than what one might expect. The reason is resources cleanup and maintaining a consistent state. Just as an example, a cancelled thread may need to unlock its mutex objects, free up memory, close file descriptors, delete named shared memory and the list can go on.
The idiom to make this work in pthread is cleanup handlers. These handlers work in a stack fashion, and can be pushed or popped at any time during run time. In case of thread cancellation by means of a call to pthread_cancel, handlers present in the stack will be automatically executed while the unwinding process occurs.
Here we have a piece of code showing how this API works:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
#include <pthread.h> #include <stddef.h> #include <stdio.h> #include <unistd.h> static pthread_mutex_t mutex; static pthread_mutex_t* mutex_ptr = NULL; static pthread_cond_t thread_state_cond; static pthread_cond_t* thread_state_cond_ptr = NULL; static int thread_state = 0; // Sync void thread_cleanup(void* args) { pthread_mutex_lock(mutex_ptr); printf("Thread: thread_state = 2.\n"); thread_state = 2; pthread_cond_broadcast(thread_state_cond_ptr); pthread_mutex_unlock(mutex_ptr); } static void thread_f(void) { int ret = -1; pthread_mutex_lock(mutex_ptr); printf("Thread: thread_state = 1.\n"); thread_state = 1; pthread_cond_broadcast(thread_state_cond_ptr); pthread_mutex_unlock(mutex_ptr); while (1) { sleep(1000); } } static void* thread_main(void* args) { pthread_cleanup_push(&thread_cleanup, NULL); thread_f(); // This should never be executed pthread_cleanup_pop(0); return NULL; } int main(void) { int ret = 0; pthread_t thread; pthread_attr_t thread_attributes; pthread_attr_t* thread_attributes_ptr = NULL; if (pthread_mutex_init(&mutex, NULL) != 0) goto error; mutex_ptr = &mutex; if (pthread_cond_init(&thread_state_cond, NULL) != 0) goto error; thread_state_cond_ptr = &thread_state_cond; if (pthread_attr_init(&thread_attributes) != 0) goto error; thread_attributes_ptr = &thread_attributes; if (pthread_create(&thread, thread_attributes_ptr, &thread_main, NULL) != 0) goto error; thread_attributes_ptr = NULL; if (pthread_attr_destroy(&thread_attributes) != 0) goto error; // Wait for thread to go deep into the call stack pthread_mutex_lock(mutex_ptr); while (thread_state != 1) pthread_cond_wait(thread_state_cond_ptr, mutex_ptr); printf("Main thread: thread_state == 1.\n"); pthread_mutex_unlock(mutex_ptr); if (pthread_cancel(thread) != 0) goto error; // Wait for thread to execute the cleanup function pthread_mutex_lock(mutex_ptr); while (thread_state != 2) pthread_cond_wait(thread_state_cond_ptr, mutex_ptr); printf("Main thread: thread_state == 2.\n"); pthread_mutex_unlock(mutex_ptr); thread_state_cond_ptr = NULL; if (pthread_cond_destroy(&thread_state_cond) != 0) goto error; mutex_ptr = NULL; if (pthread_mutex_destroy(&mutex) != 0) goto error; goto cleanup; error: ret = -1; cleanup: if (thread_attributes_ptr != NULL) pthread_attr_destroy(thread_attributes_ptr); if (thread_state_cond_ptr != NULL) pthread_cond_destroy(thread_state_cond_ptr); if (mutex_ptr != NULL) pthread_mutex_destroy(mutex_ptr); if (ret == -1) printf("Finished with errors.\n"); else printf("Finished with no errors.\n"); return ret; } |
Don’t get distracted by the number of lines because what this code does is pretty simple. The first thread, executing main, creates a second thread and waits for it to make some progress. The second thread executes thread_main, pushes a cleanup handler, calls thread_f and waits to be cancelled. The first thread will then call pthread_cancel. During the cancellation process, the pushed cleanup handler has to be executed.
pthread_cleanup_push, used to push the cleanup handler to the stack, looks like a function. But that’s not the case in Linux x86_64 -at least-. It’s a macro, and this is how the expanded assembly looks like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
0000000000400cb2 <thread_main>: 400cb2: 55 push %rbp 400cb3: 48 89 e5 mov %rsp,%rbp 400cb6: 48 81 ec 90 00 00 00 sub $0x90,%rsp 400cbd: 48 89 bd 78 ff ff ff mov %rdi,-0x88(%rbp) 400cc4: 48 c7 45 f8 06 0c 40 movq $0x400c06,-0x8(%rbp) 400ccb: 00 400ccc: 48 c7 45 f0 00 00 00 movq $0x0,-0x10(%rbp) 400cd3: 00 400cd4: 48 8d 45 80 lea -0x80(%rbp),%rax 400cd8: be 00 00 00 00 mov $0x0,%esi 400cdd: 48 89 c7 mov %rax,%rdi 400ce0: e8 eb fd ff ff callq 400ad0 <__sigsetjmp@plt> 400ce5: 89 45 ec mov %eax,-0x14(%rbp) 400ce8: 8b 45 ec mov -0x14(%rbp),%eax 400ceb: 48 98 cltq 400ced: 48 85 c0 test %rax,%rax 400cf0: 74 19 je 400d0b <thread_main+0x59> 400cf2: 48 8b 55 f0 mov -0x10(%rbp),%rdx 400cf6: 48 8b 45 f8 mov -0x8(%rbp),%rax 400cfa: 48 89 d7 mov %rdx,%rdi 400cfd: ff d0 callq *%rax 400cff: 48 8d 45 80 lea -0x80(%rbp),%rax 400d03: 48 89 c7 mov %rax,%rdi 400d06: e8 b5 fd ff ff callq 400ac0 <__pthread_unwind_next@plt> 400d0b: 48 8d 45 80 lea -0x80(%rbp),%rax 400d0f: 48 89 c7 mov %rax,%rdi 400d12: e8 19 fd ff ff callq 400a30 <__pthread_register_cancel@plt> 400d17: e8 3a ff ff ff callq 400c56 <thread_f> 400d1c: 48 8d 45 80 lea -0x80(%rbp),%rax 400d20: 48 89 c7 mov %rax,%rdi 400d23: e8 48 fd ff ff callq 400a70 <__pthread_unregister_cancel@plt> 400d28: b8 00 00 00 00 mov $0x0,%eax 400d2d: c9 leaveq 400d2e: c3 retq |
When pushing the cleanup handler, a __sigsetjmp call is made (see address 0x400ce0). Information about the context (register values) is kept in a local variable (-0x80(%rbp)). As with any setjmp/longjmp, there are two possible paths to continue execution: 1) the path taken after calling setjmp and saving a context, and 2) the path taken after a context restore (longjmp call). Which path to take will depend on the RAX register value at 0x400ce5. The first path corresponds to a non-cancellation flow and the second to a cancellation one.
Let’s analyze both flows in more detail.
Non-cancellation flow
- two local variables contain pointers to the cleanup handler function (-0x8(%rbp)) and to the clenaup handler argument (-0x10(%rbp))
- setjmp is called and returns 0x0
- the current context is saved in a local variable because it might be needed in the event of a cancellation
- __pthread_register_cancel is called
- the saved context is now part of a chain and there is a thread-local variable set to the latest context saved. See glibc’s nptl/cleanup.c for further information.
- thread_f is called, going deeper into the call stack
- __pthread_unregister_cancel is called, once returning from thread_f
- the saved context is removed from the chain and the thread-local variable is updated accordingly
Cancellation flow
- a thread calls __pthread_cancel (glibc – nptl/pthread_cancel.c)
- a SIGCANCEL signal is sent to the thread being cancelled
- the thread being cancelled handles the signal with __pthread_enable_asynccancel (glibc – sysdeps/unix/sysv/linux/x86_64/cancellation.S)
- __pthread_unwind (glibc – nptl/unwind.c) is called and the thread-local variable pointing to the last context saved is sent as an argument
- after a few calls, execution finally reaches _Unwind_ForcedUnwind_Phase2 (libgcc – libgcc/unwind.inc). The stop function is unwind_stop (glibc – ntpl/unwind.c). For every unwinded stack frame, unwind_stop will be called. Personality functions are not used at all, contrary to C++ or other languages unwinding.
- unwind_stop decides whether or not the last context saved corresponds to the frame being unwinded. If it does not, unwinding moves to the next frame. If it does, longjmp is called and execution in our example goes straight to 0x400ce5; but this time with an RAX value of 0x1.
- the cleanup handler is called at 0x400cfd
- when the cleanup handler returns, __pthread_unwind_next is called for the unwinding process to continue.