Thread cancellation is a bit trickier than what one might expect. The reason is resources cleanup and maintaining a consistent state. Just as an example, a cancelled thread may need to unlock its mutex objects, free up memory, close file descriptors, delete named shared memory and the list can go on.
The idiom to make this work in pthread is cleanup handlers. These handlers work in a stack fashion, and can be pushed or popped at any time during run time. In case of thread cancellation by means of a call to pthread_cancel, handlers present in the stack will be automatically executed while the unwinding process occurs.
Here we have a piece of code showing how this API works:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
|
#include <pthread.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
static pthread_mutex_t mutex;
static pthread_mutex_t* mutex_ptr = NULL;
static pthread_cond_t thread_state_cond;
static pthread_cond_t* thread_state_cond_ptr = NULL;
static int thread_state = 0; // Sync
void thread_cleanup(void* args) {
pthread_mutex_lock(mutex_ptr);
printf("Thread: thread_state = 2.\n");
thread_state = 2;
pthread_cond_broadcast(thread_state_cond_ptr);
pthread_mutex_unlock(mutex_ptr);
}
static void thread_f(void) {
int ret = -1;
pthread_mutex_lock(mutex_ptr);
printf("Thread: thread_state = 1.\n");
thread_state = 1;
pthread_cond_broadcast(thread_state_cond_ptr);
pthread_mutex_unlock(mutex_ptr);
while (1) {
sleep(1000);
}
}
static void* thread_main(void* args) {
pthread_cleanup_push(&thread_cleanup, NULL);
thread_f();
// This should never be executed
pthread_cleanup_pop(0);
return NULL;
}
int main(void) {
int ret = 0;
pthread_t thread;
pthread_attr_t thread_attributes;
pthread_attr_t* thread_attributes_ptr = NULL;
if (pthread_mutex_init(&mutex, NULL) != 0)
goto error;
mutex_ptr = &mutex;
if (pthread_cond_init(&thread_state_cond, NULL) != 0)
goto error;
thread_state_cond_ptr = &thread_state_cond;
if (pthread_attr_init(&thread_attributes) != 0)
goto error;
thread_attributes_ptr = &thread_attributes;
if (pthread_create(&thread, thread_attributes_ptr, &thread_main, NULL) != 0)
goto error;
thread_attributes_ptr = NULL;
if (pthread_attr_destroy(&thread_attributes) != 0)
goto error;
// Wait for thread to go deep into the call stack
pthread_mutex_lock(mutex_ptr);
while (thread_state != 1)
pthread_cond_wait(thread_state_cond_ptr, mutex_ptr);
printf("Main thread: thread_state == 1.\n");
pthread_mutex_unlock(mutex_ptr);
if (pthread_cancel(thread) != 0)
goto error;
// Wait for thread to execute the cleanup function
pthread_mutex_lock(mutex_ptr);
while (thread_state != 2)
pthread_cond_wait(thread_state_cond_ptr, mutex_ptr);
printf("Main thread: thread_state == 2.\n");
pthread_mutex_unlock(mutex_ptr);
thread_state_cond_ptr = NULL;
if (pthread_cond_destroy(&thread_state_cond) != 0)
goto error;
mutex_ptr = NULL;
if (pthread_mutex_destroy(&mutex) != 0)
goto error;
goto cleanup;
error:
ret = -1;
cleanup:
if (thread_attributes_ptr != NULL)
pthread_attr_destroy(thread_attributes_ptr);
if (thread_state_cond_ptr != NULL)
pthread_cond_destroy(thread_state_cond_ptr);
if (mutex_ptr != NULL)
pthread_mutex_destroy(mutex_ptr);
if (ret == -1)
printf("Finished with errors.\n");
else
printf("Finished with no errors.\n");
return ret;
}
|
Don’t get distracted by the number of lines because what this code does is pretty simple. The first thread, executing main, creates a second thread and waits for it to make some progress. The second thread executes thread_main, pushes a cleanup handler, calls thread_f and waits to be cancelled. The first thread will then call pthread_cancel. During the cancellation process, the pushed cleanup handler has to be executed.
pthread_cleanup_push, used to push the cleanup handler to the stack, looks like a function. But that’s not the case in Linux x86_64 -at least-. It’s a macro, and this is how the expanded assembly looks like:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
0000000000400cb2 <thread_main>:
400cb2: 55 push %rbp
400cb3: 48 89 e5 mov %rsp,%rbp
400cb6: 48 81 ec 90 00 00 00 sub $0x90,%rsp
400cbd: 48 89 bd 78 ff ff ff mov %rdi,-0x88(%rbp)
400cc4: 48 c7 45 f8 06 0c 40 movq $0x400c06,-0x8(%rbp)
400ccb: 00
400ccc: 48 c7 45 f0 00 00 00 movq $0x0,-0x10(%rbp)
400cd3: 00
400cd4: 48 8d 45 80 lea -0x80(%rbp),%rax
400cd8: be 00 00 00 00 mov $0x0,%esi
400cdd: 48 89 c7 mov %rax,%rdi
400ce0: e8 eb fd ff ff callq 400ad0 <__sigsetjmp@plt>
400ce5: 89 45 ec mov %eax,-0x14(%rbp)
400ce8: 8b 45 ec mov -0x14(%rbp),%eax
400ceb: 48 98 cltq
400ced: 48 85 c0 test %rax,%rax
400cf0: 74 19 je 400d0b <thread_main+0x59>
400cf2: 48 8b 55 f0 mov -0x10(%rbp),%rdx
400cf6: 48 8b 45 f8 mov -0x8(%rbp),%rax
400cfa: 48 89 d7 mov %rdx,%rdi
400cfd: ff d0 callq *%rax
400cff: 48 8d 45 80 lea -0x80(%rbp),%rax
400d03: 48 89 c7 mov %rax,%rdi
400d06: e8 b5 fd ff ff callq 400ac0 <__pthread_unwind_next@plt>
400d0b: 48 8d 45 80 lea -0x80(%rbp),%rax
400d0f: 48 89 c7 mov %rax,%rdi
400d12: e8 19 fd ff ff callq 400a30 <__pthread_register_cancel@plt>
400d17: e8 3a ff ff ff callq 400c56 <thread_f>
400d1c: 48 8d 45 80 lea -0x80(%rbp),%rax
400d20: 48 89 c7 mov %rax,%rdi
400d23: e8 48 fd ff ff callq 400a70 <__pthread_unregister_cancel@plt>
400d28: b8 00 00 00 00 mov $0x0,%eax
400d2d: c9 leaveq
400d2e: c3 retq
|
When pushing the cleanup handler, a __sigsetjmp call is made (see address 0x400ce0). Information about the context (register values) is kept in a local variable (-0x80(%rbp)). As with any setjmp/longjmp, there are two possible paths to continue execution: 1) the path taken after calling setjmp and saving a context, and 2) the path taken after a context restore (longjmp call). Which path to take will depend on the RAX register value at 0x400ce5. The first path corresponds to a non-cancellation flow and the second to a cancellation one.
Let’s analyze both flows in more detail.
Non-cancellation flow
- two local variables contain pointers to the cleanup handler function (-0x8(%rbp)) and to the clenaup handler argument (-0x10(%rbp))
-
setjmp is called and returns 0x0
- the current context is saved in a local variable because it might be needed in the event of a cancellation
-
__pthread_register_cancel is called
- the saved context is now part of a chain and there is a thread-local variable set to the latest context saved. See glibc’s nptl/cleanup.c for further information.
- thread_f is called, going deeper into the call stack
-
__pthread_unregister_cancel is called, once returning from thread_f
- the saved context is removed from the chain and the thread-local variable is updated accordingly
Cancellation flow
- a thread calls __pthread_cancel (glibc – nptl/pthread_cancel.c)
- a SIGCANCEL signal is sent to the thread being cancelled
- the thread being cancelled handles the signal with __pthread_enable_asynccancel (glibc – sysdeps/unix/sysv/linux/x86_64/cancellation.S)
- __pthread_unwind (glibc – nptl/unwind.c) is called and the thread-local variable pointing to the last context saved is sent as an argument
- after a few calls, execution finally reaches _Unwind_ForcedUnwind_Phase2 (libgcc – libgcc/unwind.inc). The stop function is unwind_stop (glibc – ntpl/unwind.c). For every unwinded stack frame, unwind_stop will be called. Personality functions are not used at all, contrary to C++ or other languages unwinding.
- unwind_stop decides whether or not the last context saved corresponds to the frame being unwinded. If it does not, unwinding moves to the next frame. If it does, longjmp is called and execution in our example goes straight to 0x400ce5; but this time with an RAX value of 0x1.
- the cleanup handler is called at 0x400cfd
- when the cleanup handler returns, __pthread_unwind_next is called for the unwinding process to continue.