Building an Async HTTPS Client with Goblins and Fibers › Fibers-Only HTTP Client [#253]

March 28, 2026
Jinser Kafka

Now we can create a wrapper that opens a non-blocking socket and passes it to the HTTP procedure:

(define (wrap-http-nonblocking http-proc)
  (λ (uri . args)
    (let ([port (open-socket-for-uri
                  uri
                  #:configure-socket use-nonblocking-i/o!)])
      (apply http-proc uri #:port port args))))

This higher-order function:

Takes an HTTP procedure (like http-request)
Returns a new function that opens a non-blocking socket
Passes that socket to the original procedure

But spawn-fiber doesn't return the result—it returns the fiber object. We need channels to communicate results back. Here's a complete working example that demonstrates true concurrency:

(use-modules (fibers))
(use-modules (fibers channels))
(use-modules (web response))
(use-modules ((web client) #:hide (open-socket-for-uri)))

;;; Borrow open-socket-for-uri from guile-websocket's internal module
(define open-socket-for-uri
  (@@ (web socket client) open-socket-for-uri))

;;; Set O_NONBLOCK flag on a port
(define (use-nonblocking-i/o! port)
  (fcntl port F_SETFL
         (logior O_NONBLOCK (fcntl port F_GETFL))))

;;; Wrap an HTTP procedure to use non-blocking sockets
(define (wrap-http-nonblocking http-proc)
  (λ (uri . args)
    (let ([port (open-socket-for-uri
                  uri
                  #:configure-socket use-nonblocking-i/o!)])
      (apply http-proc uri #:port port args))))

;;; Make a delayed request and send result to channel
(define (make-delayed-request delay-seconds channel)
  (spawn-fiber
    (λ ()
      (display (format #f "[~a] Starting request with ~a second delay~%"
                       (strftime "%H:%M:%S" (localtime (current-time)))
                       delay-seconds))
      (define-values (response body)
        ((wrap-http-nonblocking http-request)
         (string-append "https://httpbin.org/delay/"
                       (number->string delay-seconds))))
      (display (format #f "[~a] Completed ~a second delay request, status: ~a~%"
                       (strftime "%H:%M:%S" (localtime (current-time)))
                       delay-seconds
                       (response-code response)))
      (put-message channel (cons delay-seconds response)))))

;;; Run concurrent requests with different delays
(define result-channel (make-channel))

(run-fibers
  (λ ()
    (display "Starting 3 concurrent requests with delays of 5s, 2s, and 3s...")
    (newline)
    (display (format #f "[~a] Main fiber starting~%~%"
                    (strftime "%H:%M:%S" (localtime (current-time)))))

    ;; Spawn three fibers
    (make-delayed-request 5 result-channel)
    (make-delayed-request 2 result-channel)
    (make-delayed-request 3 result-channel)

    ;; Collect results - they'll arrive in completion order (2s, 3s, 5s)
    (let loop ([n 0])
      (when (< n 3)
        (define result (get-message result-channel))
        (display (format #f "[~a] Received result from ~as delay request~%"
                        (strftime "%H:%M:%S" (localtime (current-time)))
                        (car result)))
        (loop (1+ n))))

    (display "\nAll requests completed!"))
  #:drain? #t)

When you run this, you'll see the 2-second request complete first, then 3-second, then 5-second—proving the fibers are truly concurrent. The total execution time is ~5 seconds, not 10 seconds.

Context

Building an Async HTTPS Client with Goblins and Fibers [jsr-001C]

March 28, 2026
Jinser Kafka

Task. This article is primarily written by LLM with human supervision and review. It remains a work in progress and is subject to further modification and updates. [#247]

March 28, 2026
Jinser Kafka

This is a walkthrough of building an asynchronous HTTP client in Guile Scheme. We start with Fibers' Concurrent ML (CML) primitives, get async HTTP working there first, and then integrate it into Goblins' actor framework.

The goal: Create an HTTPS client that can make non-blocking requests without freezing the system, first using Fibers alone, then integrated with Goblins' actor model. While HTTP would work similarly, HTTPS presents a unique challenge due to TLS handling.

The Problem with Blocking I/O [#248]

March 28, 2026
Jinser Kafka

Guile's standard (web client) module provides http-request, http-get, etc., but these are synchronous. They block until the response is received. In any concurrent system—whether simple fibers or full actor frameworks—blocking operations freeze the entire execution context.

We needed a way to perform HTTP requests that:

Don't block other fibers from running
Work with Fibers' cooperative scheduling
Can later be integrated into Goblins' actor model

Understanding Fibers [#249]

March 28, 2026
Jinser Kafka

Fibers, developed by Andy Wingo, implements Concurrent ML (CML) as described in his article A New Concurrent ML. Fibers builds upon the theoretical foundation laid by John Reppy in his seminal papers Synchronous Operations as First-Class Values and CML: A Higher Concurrent Language. Unlike preemptive threading, CML-style concurrency is cooperative—fibers explicitly yield control at synchronization points.

The key insight of CML is that synchronous operations are abstractions over potential synchronizations—much like a lambda expression is an abstraction over an expression that may be evaluated later. An operation is a value that can be stored, passed around, and composed. You can perform (or in CML terminology, synchronize) an operation zero times, once, or multiple times—only then does the actual synchronization occur.

Fibers uses the term operation rather than CML's original event, and perform-operation rather than synchronize, but the concepts are identical. Operations can be combined using combinators like choice-operation (select from multiple operations) and wrap-operation (attach continuations to operations).

Why Standard HTTP Won't Work with Fibers [#250]

March 28, 2026
Jinser Kafka

Simply wrapping http-request in a fiber isn't sufficient. Here's the critical insight: Fibers' cooperative scheduling only works when operations can yield control.

;; This doesn't work!
(spawn-fiber
  (λ ()
    (http-request "https://example.com")))

In Fibers terms:

Standard blocking I/O is not a Fibers operation—it cannot be composed, cannot be selected among alternatives using choice-operation, and most importantly, cannot yield control
When read() blocks on a socket with no data, the fiber blocks at the OS level
The Fibers scheduler has no visibility into this blocking—other fibers cannot run until the I/O completes
For cooperative multitasking to work, I/O must itself be expressed as Fibers operations, allowing the fiber to register interest ("notify me when data arrives") and yield immediately

Without non-blocking sockets, spawn-fiber is merely wrapping a blocking operation in a fiber—no actual concurrency is achieved. The fiber blocks, and any other fibers waiting to run are stalled.

The Socket Problem [#251]

March 28, 2026
Jinser Kafka

Guile's standard (web client) module uses GnuTLS for HTTPS connections. The problem: GnuTLS wraps the underlying socket in a TLS port before we can configure it, preventing us from setting the O_NONBLOCK flag required by Fibers. As the Guile Reference Manual notes, "Any port type defined by an extension in C may not be suspendable, for example SSL ports from GnuTLS."

This only affects HTTPS. Plain HTTP works fine—the raw socket remains accessible. But in practice, we need HTTPS, and the GnuTLS layer encapsulates the socket before we can configure it for non-blocking I/O.

The solution comes from guile-websocket, which faced the same issue. They provide a patched open-socket-for-uri that accepts a #:configure-socket callback—a hook that runs after the socket is created but before TLS wrapping occurs. This is the critical difference: Guile's standard library doesn't provide this hook. Since this function isn't exported from guile-websocket's public API, we access it via Guile's @@ operator:

;;; Borrow from guile-websocket's internal (web socket client) module.
;;; This version provides #:configure-socket, a hook called before TLS wrapping.
(define open-socket-for-uri
  (@@ (web socket client) open-socket-for-uri))

Making Sockets Non-Blocking [#252]

March 28, 2026
Jinser Kafka

The Fibers documentation explains the requirements for ports to work with its cooperative scheduling:

The port has to either never block, or support non-blocking I/O. Since Guile version 3.0.10, this is the case for all ports except those that use libreadline. Any port type defined by an extension in C may not be suspendable, however, for example SSL ports from GnuTLS.

We pass this function to guile-websocket's #:configure-socket hook, which calls it after socket creation but before TLS wrapping:

(define (use-nonblocking-i/o! port)
  (fcntl port F_SETFL
         (logior O_NONBLOCK (fcntl port F_GETFL))))

This function applies the standard fcntl incantation:

Gets the current flags with F_GETFL
Bitwise-ORs O_NONBLOCK with the existing flags
Sets the new flags with F_SETFL

Fibers-Only HTTP Client [#253]

March 28, 2026
Jinser Kafka

Now we can create a wrapper that opens a non-blocking socket and passes it to the HTTP procedure:

(define (wrap-http-nonblocking http-proc)
  (λ (uri . args)
    (let ([port (open-socket-for-uri
                  uri
                  #:configure-socket use-nonblocking-i/o!)])
      (apply http-proc uri #:port port args))))

This higher-order function:

Takes an HTTP procedure (like http-request)
Returns a new function that opens a non-blocking socket
Passes that socket to the original procedure

But spawn-fiber doesn't return the result—it returns the fiber object. We need channels to communicate results back. Here's a complete working example that demonstrates true concurrency:

(use-modules (fibers))
(use-modules (fibers channels))
(use-modules (web response))
(use-modules ((web client) #:hide (open-socket-for-uri)))

;;; Borrow open-socket-for-uri from guile-websocket's internal module
(define open-socket-for-uri
  (@@ (web socket client) open-socket-for-uri))

;;; Set O_NONBLOCK flag on a port
(define (use-nonblocking-i/o! port)
  (fcntl port F_SETFL
         (logior O_NONBLOCK (fcntl port F_GETFL))))

;;; Wrap an HTTP procedure to use non-blocking sockets
(define (wrap-http-nonblocking http-proc)
  (λ (uri . args)
    (let ([port (open-socket-for-uri
                  uri
                  #:configure-socket use-nonblocking-i/o!)])
      (apply http-proc uri #:port port args))))

;;; Make a delayed request and send result to channel
(define (make-delayed-request delay-seconds channel)
  (spawn-fiber
    (λ ()
      (display (format #f "[~a] Starting request with ~a second delay~%"
                       (strftime "%H:%M:%S" (localtime (current-time)))
                       delay-seconds))
      (define-values (response body)
        ((wrap-http-nonblocking http-request)
         (string-append "https://httpbin.org/delay/"
                       (number->string delay-seconds))))
      (display (format #f "[~a] Completed ~a second delay request, status: ~a~%"
                       (strftime "%H:%M:%S" (localtime (current-time)))
                       delay-seconds
                       (response-code response)))
      (put-message channel (cons delay-seconds response)))))

;;; Run concurrent requests with different delays
(define result-channel (make-channel))

(run-fibers
  (λ ()
    (display "Starting 3 concurrent requests with delays of 5s, 2s, and 3s...")
    (newline)
    (display (format #f "[~a] Main fiber starting~%~%"
                    (strftime "%H:%M:%S" (localtime (current-time)))))

    ;; Spawn three fibers
    (make-delayed-request 5 result-channel)
    (make-delayed-request 2 result-channel)
    (make-delayed-request 3 result-channel)

    ;; Collect results - they'll arrive in completion order (2s, 3s, 5s)
    (let loop ([n 0])
      (when (< n 3)
        (define result (get-message result-channel))
        (display (format #f "[~a] Received result from ~as delay request~%"
                        (strftime "%H:%M:%S" (localtime (current-time)))
                        (car result)))
        (loop (1+ n))))

    (display "\nAll requests completed!"))
  #:drain? #t)

When you run this, you'll see the 2-second request complete first, then 3-second, then 5-second—proving the fibers are truly concurrent. The total execution time is ~5 seconds, not 10 seconds.

From Local to Distributed [#254]

March 28, 2026
Jinser Kafka

While Fibers gives us the low-level mechanism for non-blocking I/O, our ultimate goal is a distributed HTTP client—one that can communicate with other objects across a network. This requires more than just local concurrency primitives.

The Distributed Challenge: We need an architecture that can seamlessly handle both local and remote communication. Fibers provides cooperative multitasking within a single process, but it doesn't give us:

Network transparency: The ability to invoke HTTP operations on remote objects as if they were local
Object capability security: A security model based on reference access rather than identity
Distributed promise pipelining: The ability to compose async operations across network boundaries
Location independence: Code that works the same whether components are in the same process or across the globe

This is where Goblins comes in. Goblins is not just an actor library—it's a distributed object programming environment. While it uses actors (objects that communicate via message passing) as its foundation, its true power lies in the OCapN (Object Capability Network) protocol. Actors are the means; distributed, secure, networked communication is the end.

By wrapping our HTTP client in Goblins actors, we gain the ability to:

Use the same HTTP client interface locally or remotely
Leverage sophisticated promise pipelining for async composition
Build on object capability security guarantees
Prepare for a distributed architecture from day one

Goblins Actor Implementation [#255]

March 28, 2026
Jinser Kafka

Goblins builds on top of Fibers, providing a distributed object environment. The fibrous macro combines spawn-fiber with Goblins' sophisticated promise system—called vows. Unlike simple Fibers channels, vows support distributed promise pipelining: you can compose async operations across network boundaries.

Here's the complete Goblins implementation with the same httpbin delay test:

(use-modules (goblins))
(use-modules (goblins vat))
(use-modules (goblins actor-lib methods))
(use-modules (fibers conditions))
(use-modules (web response))
(use-modules ((web client) #:hide (open-socket-for-uri)))

;;; Borrow open-socket-for-uri from guile-websocket
(define open-socket-for-uri (@@ (web socket client) open-socket-for-uri))

(define (use-nonblocking-i/o! port)
  (fcntl port F_SETFL
         (logior O_NONBLOCK (fcntl port F_GETFL))))

(define (wrap-http-nonblocking http-proc)
  (λ (uri args)
    (let ([port (open-socket-for-uri uri #:configure-socket use-nonblocking-i/o!)])
      (apply http-proc uri #:port port args))))

;;; Define the HTTP client actor
(define-actor (^http-client _bcom)
  #:self self
  (methods
    [(request uri . args)
     (fibrous (call-with-values (λ () ((wrap-http-nonblocking http-request) uri args))
                                cons))]
    [(get     uri . args) (apply $ self 'request uri #:method 'GET     args)]
    [(head    uri . args) (apply $ self 'request uri #:method 'HEAD    args)]
    [(post    uri . args) (apply $ self 'request uri #:method 'POST    args)]
    [(put     uri . args) (apply $ self 'request uri #:method 'PUT     args)]
    [(delete  uri . args) (apply $ self 'request uri #:method 'DELETE  args)]
    [(trace   uri . args) (apply $ self 'request uri #:method 'TRACE   args)]
    [(options uri . args) (apply $ self 'request uri #:method 'OPTIONS args)]))

;;; Demonstrate async behavior with concurrent HTTP requests
(define (run-demo)
  (define done? (make-condition))
  (define completed-count 0)
  (define total-requests 3)

  (define (check-complete!)
    (set! completed-count (1+ completed-count))
    (when (= completed-count total-requests)
      (signal-condition! done?)))

  (define vat (spawn-vat))

  (with-vat vat
    (define http-client (spawn ^http-client))

    (display "Starting 3 concurrent requests with delays of 5s, 2s, and 3s...")
    (newline)
    (display (format #f "[~a] Main vat starting" 
                     (strftime "%H:%M:%S" (localtime (current-time)))))
    (newline)
    (newline)

    ;; Request 1: 5 second delay
    (on (<- http-client 'get "https://httpbin.org/delay/5")
        (λ (result)
          (display (format #f "[~a] Completed 5s delay request, status: ~a"
                           (strftime "%H:%M:%S" (localtime (current-time)))
                           (response-code (car result))))
          (newline)
          (check-complete!)))

    ;; Request 2: 2 second delay (will complete first)
    (on (<- http-client 'get "https://httpbin.org/delay/2")
        (λ (result)
          (display (format #f "[~a] Completed 2s delay request, status: ~a"
                           (strftime "%H:%M:%S" (localtime (current-time)))
                           (response-code (car result))))
          (newline)
          (check-complete!)))

    ;; Request 3: 3 second delay (will complete second)
    (on (<- http-client 'get "https://httpbin.org/delay/3")
        (λ (result)
          (display (format #f "[~a] Completed 3s delay request, status: ~a"
                           (strftime "%H:%M:%S" (localtime (current-time)))
                           (response-code (car result))))
          (newline)
          (check-complete!))))

  ;; Wait for all requests to complete
  ;; Note: on returns immediately, so we need done? to prevent exit
  (wait done?)

  (vat-halt! vat)

  (display "")
  (display (format #f "[~a] All requests completed!" 
                   (strftime "%H:%M:%S" (localtime (current-time)))))
  (newline)
  'done)

;; Run the demo
(run-demo)

Key differences from the Fibers version:

We use on handlers instead of explicit channels—Goblins manages the async flow
The <- operator sends messages to actors, which can be local or remote
We use done? condition to wait for completion because on returns immediately
The actor can be transparently moved to another machine via OCapN without code changes

The request method uses fibrous to leverage Fibers for non-blocking I/O while returning a Goblins vow. This vow can be pipelined with other async operations, even across network boundaries. The call-with-values handles Guile's multiple return values, and cons packages them for distributed messaging.

Future Improvements [#256]

March 28, 2026
Jinser Kafka

Some ideas for extending this:

Connection pooling: Reuse connections for multiple requests to the same host
Timeout handling: Add timeouts to prevent waiting forever
Request/response interceptors: Add middleware for logging, retries, etc.

But the foundation is solid—a clean, composable async HTTP client that plays nicely with Goblins' actor model and can scale from local to distributed.