Request hangs despite :connection-timeout

[erjoalgo]

How can I stop the following request from hanging indefinitely on SBCL?

(drakma:http-request "https://health.usnews.com/doctors/carolyn-connelly-544761" :connection-timeout 5)
``

[phoe]

I can reproduce this issue. It seems like Drakma has established a connection but fails to receive any incoming data.

REPL:

CL-USER> (setf drakma:*header-stream* *standard-output*)
#<SYNONYM-STREAM :SYMBOL SWANK::*CURRENT-STANDARD-OUTPUT* {101B538553}>
CL-USER> (drakma:http-request "https://health.usnews.com/doctors/carolyn-connelly-544761" :connection-timeout 5)
GET /doctors/carolyn-connelly-544761 HTTP/1.1
Host: health.usnews.com
User-Agent: Drakma/2.0.7 (SBCL 2.1.0; Linux; 5.10.0-1-amd64; http://weitz.de/drakma/)
Accept: */*
Connection: close

Backtrace:

Interrupt from Emacs
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [CONTINUE] Continue from break.
 1: [RETRY] Retry SLIME REPL evaluation request.
 2: [*ABORT] Return to SLIME's top level.
 3: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1010031EC3}>)

Backtrace:
  0: ("bogus stack frame")
  1: (SB-SYS:WAIT-UNTIL-FD-USABLE 3 :INPUT NIL T)
  2: ((:METHOD STREAM-READ-BYTE (CL+SSL::SSL-STREAM)) #<CL+SSL::SSL-STREAM for 3>) [fast-method]
  3: (READ-BYTE #<CL+SSL::SSL-STREAM for 3> NIL :EOF)
  4: ((:METHOD STREAM-READ-BYTE (CHUNGA:CHUNKED-INPUT-STREAM)) #<CHUNGA:CHUNKED-IO-STREAM {1008163BF3}>) [fast-method]
  5: (READ-BYTE #<CHUNGA:CHUNKED-IO-STREAM {1008163BF3}> NIL NIL)
  6: ((:METHOD FLEXI-STREAMS::READ-BYTE* (FLEXI-STREAMS:FLEXI-INPUT-STREAM)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}>) [fast-method]
  7: ((:METHOD STREAM-READ-BYTE (FLEXI-STREAMS:FLEXI-INPUT-STREAM)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}>) [fast-method]
  8: ((SB-PCL::EMF STREAM-READ-BYTE) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}>)
  9: (READ-BYTE #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}> NIL NIL)
 10: (CHUNGA:READ-CHAR* #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}> NIL NIL)
 11: (CHUNGA:READ-LINE* #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}> NIL)
 12: (DRAKMA::READ-STATUS-LINE #<FLEXI-STREAMS:FLEXI-IO-STREAM {1008167733}> NIL)
 13: ((LABELS DRAKMA::FINISH-REQUEST :IN DRAKMA:HTTP-REQUEST) NIL NIL)
 14: (DRAKMA:HTTP-REQUEST #<PURI:URI https://health.usnews.com/doctors/carolyn-connelly-544761> :CONNECTION-TIMEOUT 5)
 15: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DRAKMA:HTTP-REQUEST "https://health.usnews.com/doctors/carolyn-connelly-544761" :CONNECTION-TIMEOUT 5) #<NULL-LEXENV>)
 16: (EVAL (DRAKMA:HTTP-REQUEST "https://health.usnews.com/doctors/carolyn-connelly-544761" :CONNECTION-TIMEOUT 5))
 --more--

[zellerin]

From what I have observed in past (e.g., Hunchentoot discussion on similar topic), two things are involved:

1. the code for sbcl to handle timeouts during read is missing. If you put to request.lisp after openmcl deadline code something like

   #+sbcl
	      (when (and (null stream) connection-timeout)
		(setf (sb-impl::fd-stream-timeout http-stream)
		      (coerce connection-timeout 'single-float)))

the timeout during read would work on http, e.g.,

(time (ignore-errors (drakma:http-request "http://health.usnews.com/doctors/carolyn-connelly-544761" :connection-timeout 5)))

times out after some 5 secs.

Proper fix would use different keyword, of course.

2. The timeouts do not propagate well to ssl with current code. One solution is to use bio callbacks in cl+ssl; in ideal world, setting cl+ssl::*default-unwrap-stream-p* to nil should make timeouts for ssl to work, but drakma helpfully unwraps FD itself, so code would have to be changed anyway:

--- drakma-v2.0.8/util.lisp
+++ drakma-v2.0.8/util.lisp
@@ -336,7 +336,8 @@
                                                        :default))))
     (cl+ssl:with-global-context (ctx)
       (cl+ssl:make-ssl-client-stream
-       (cl+ssl:stream-fd s)
+       s
+       :unwrap-stream-p nil ; this requires recent cl+ssl, if not available just set the variable mentioned above
        :verify verify
        :hostname hostname
        :close-callback (lambda ()

After these changes I can see your call time out.

I do not know if it makes sense to try to push these changes as a patch to repo - I vaguely recall some discussion in not so recent past (~10 years ago) that this should be stuff of the libraries used, and patch on timeout for sbcl refused.

[avodonosov]

Try wrapping the drakma invocation into (sb-sys:with-deadline ...), should work.

If so, you can drop the :connection-timeout parameter. Something like:

(sb-sys:with-deadline (:seconds 13)
    (drakma:http-request "https://health.usnews.com/doctors/carolyn-connelly-544761"))

(I haven't tested this)

[phoe]

I can still reproduce this on SBCL 2.2.2 and the recent quicklisp dist. I have no idea if the issue lies on Drakma side or the CL+SSL side.

But timeouts should be completely unnecessary because that web server responds just fine. For whatever reason CL+SSL seems unable to read a byte from the stream, whereas curl can read this just fine:

$ curl "https://health.usnews.com/doctors/carolyn-connelly-544761"
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;health&#46;usnews&#46;com&#47;doctors&#47;carolyn&#45;connelly&#45;544761" on this server.<P>
Reference&#32;&#35;18&#46;6e645e68&#46;1649957515&#46;2ee9b6ad
</BODY>
</HTML>

Edit: Reproduced without Drakma, moved to cl-plus-ssl/cl-plus-ssl#156

[zellerin]

It looks like that curl appears to use HTTP/2.0, try to run it with -v. If you direct it to use plain old http, e.g., with --http1.1 parameter, it hangs as well.

This might be relevant:
https://community.akamai.com/customers/s/question/0D54R00007GjCANSA3/why-does-akamai-edge-services-sometime-just-not-send-any-response-leaving-the-connection-to-timeout?language=en_US

In cases where a request comes over HTTP2 and it’s from an IP address to which you’ve assigned the tarpit action, Bot Manager returns a 403 deny response instead. Multiple client requests could get multiplexed over the same HTTP2 connection, so a tarpit action would inadvertently affect all clients sharing the session. Bot Manager errs on the side of caution and instead denies the offending IP address when the request is over HTTP2.

I am relatively sure that Drakma does not speak http/2 at the moment; this is a binary protocol and AFAIK requires alpn support in cl+ssl, which has been added very recently. Whether it should and will might be a discussion worthy a separate issue, as this one was about timeouts.