[Info-vax] Problems with multithreaded calls to libCURL

[gezelter at rlgsc.com]

On Thursday, March 7, 2019 at 6:09:39 PM UTC-5, Jim Duff wrote:
> Versions:
> 
> OpenVMS 8.4 on IA64
> HP C V7.2-001:
> libCURL: 7.4.7
> 
> Nearly seven years ago, I wrote some code to utilise libCURL to perform
> FTP for a document routing system.  The code was single threaded, and it
> quickly became the event driven successor for what was then a polling
> system based on CIFS.
> 
> Today, the system ships hundreds of thousands of documents a day off the
> central ERP system.  All usually goes well, but there is at least one
> problem that can seriously impact document delivery times.  This occurs
> when an FTP server at a target machine is unavailable.
> 
> When this occurs, with the default setting of CURLOPT_CONNECTTIMEOUT, a
> 5 minute delay can occur as the library waits for the connection to
> complete.  Easy fix says I, just set CURLOPT_CONNECTIMEOUT to a smaller
> value with a call like
> 
> res = curl_easy_setopt (handle, CURLOPT_CONNECTTIMEOUT, 30L);
> 
> for a 30 second timeout.
> 
> Except this appears to have no effect.
> 
> After double checking that no later version of libCURL is currently
> available for OpenVMS, I embarked on multithreading the code so that
> there would be one worker thread per server and target directory
> combination.  That way, if an FTP server was unavailable, only documents
> destined for that server would be impacted by the connect timeout issue
> as only the affected thread(s) would stall.
> 
> To my consternation, on stress testing the multithreaded code, I'm
> occasionally getting an access violation in the depths of libCURL:
> 
> %SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual
> address=000000000000
> 0028, PC=FFFFFFFF84088EB0, PS=0000001B
> %TRACE-F-TRACEBACK, symbolic stack dump follows
> image     module    routine               line      rel PC           abs PC
> LIBRTL  LIB$$UNWIND  restoreReg          22380 00000000000007A0
> FFFFFFFF84088EB0
> LIBRTL  LIB$$UNWIND  step_final_phase    25871 0000000000010652
> FFFFFFFF84098D62
> LIBRTL  LIB$$UNWIND  UC_step             24860 000000000000C682
> FFFFFFFF84094D92
> LIBRTL  LIB$$UNWIND  UC_currentContext
>                                          21987 0000000000000602
> FFFFFFFF84088D12
> LIBRTL  LIB$IPF_CALLING_STANDARD  lib$i64_get_curr_invo_context
>                                          19476 00000000000009A2
> FFFFFFFF8407FAB2
> DECC$SHR  C$SETJMP  sigsetjmp            17136 00000000000008B2
> FFFFFFFF84B6DBE2
> GNV$LIBCURL  hostip  Curl_resolv_timeout
>                                          82084 0000000000001382
> 000000000020E322
> GNV$LIBCURL  url  create_conn            92703 00000000000145E2
> 00000000001FFDD2
> GNV$LIBCURL  url  Curl_connect           93456 0000000000014D12
> 0000000000200502
> GNV$LIBCURL  multi  multi_runsingle      83372 0000000000003002
> 00000000001DCF62
> GNV$LIBCURL  multi  curl_multi_perform
>                                          84076 0000000000006C52
> 00000000001E0BB2
> GNV$LIBCURL  easy  curl_easy_perform     87297 0000000000000C52
> 00000000001CCC52
> FTP_SENDER  FTP_SENDER  do_ftp           62939 00000000000027C2
> 00000000000227C2
> FTP_SENDER  FTP_SENDER  worker           63195 00000000000047D2
> 00000000000247D2
> PTHREAD$RTL  THD_THREAD  thdBase        245274 0000000000005C52
> FFFFFFFF844D4ED2
> PTHREAD$RTL                                  0 0000000000054012
> FFFFFFFF844AE012
> PTHREAD$RTL  THD_INIT  pthread_main     244961 0000000000000492
> FFFFFFFF8448A492
>                                              0 FFFFFFFF80B026D2
> FFFFFFFF80B026D2
> DCL                                          0 000000000007D072
> 000000007AE21072
> %TRACE-I-END, end of TRACE stack dump
> 
> Enclosing the calls to curl_easy_perform () in a mutex makes the access
> violation go away, making me think that libCURL's claim to being thread
> safe is not so rock solid.
> 
> Reading https://curl.haxx.se/libcurl/c/threadsafe.html I see there are
> some exceptions to its thread safe claim, so I duly implemented them:
> 
> 1) don't share handles between threads.  I wasn't doing this in the
> first place.
> 2) I'm not using the share interface.
> 3) disable signals with CURLOPT_NOSIGNAL set to 1L
> 4) call curl_global_init () in the main thread before creating
> additional threads.
> 5) don't use CURLOPT_DNS_USE_GLOBAL_CACHE.  I wasn't.
> 
> I'd love to debug this myself, but I have so much on my plate that I
> have no time to do so.
> 
> If anyone has any ideas, or has implemented a multithreaded system
> employing libCURL, I'd love to hear from you.
> 
> Jim

Jim,

I would be suspicious of whether the socket library is completely thread-safe. I have done what you are trying to do, with other protocols, and event-driven (ASTs), but I used QIO for all of the TCP and related functions. 

- Bob Gezelter, http://www.rlgsc.com