How does Django’s StreamingHttpResponse work, exactly?

This post tries to explain just what goes on when you use Django’s StreamingHttpResponse.

I will discuss what happens in your Django application, what happens at the Python Web Server Gateway Interface (WSGI) layer, and look at some examples.

This content is also available as a README with an example Django project.

So, what even is a StreamingHttpResponse?

Most Django responses use HttpResponse. At a high level, this means that the body of the response is built in memory and sent to the HTTP client in a single piece.

Here’s a short example of using HttpResponse:

    def my_view(request):
        message = 'Hello, there!'
        response =  HttpResponse(message)
        response['Content-Length'] = len(message)

        return response

A StreamingHttpResponse, on the other hand, is a response whose body is sent to the client in multiple pieces, or “chunks.”

Here’s a short example of using StreamingHttpResponse:

    def hello():
        yield 'Hello,'
        yield 'there!'

    def my_view(request):
        # NOTE: No Content-Length header!
        return StreamingHttpResponse(hello)

You can read more about how to use these two classes in Django’s documentation. The interesting part is what happens next — after you return the response.

When would you use a StreamingHttpResponse?

But before we talk about what happens after you return the response, let us digress for a moment: why would you even use a StreamingHttpResponse?

One of the best use cases for streaming responses is to send large files, e.g. a large CSV file.

With an HttpResponse, you would typically load the entire file into memory (produced dynamically or not) and then send it to the client. For a large file, this costs memory on the server and “time to first byte” (TTFB) sent to the client.

With a StreamingHttpResponse, you can load parts of the file into memory, or produce parts of the file dynamically, and begin sending these parts to the client immediately. Crucially, there is no need to load the entire file into memory.

A quick note about WSGI

Now we’re approaching the part of our journey that lies just beyond most Django developers’ everyday experience of working with Django’s response classes.

Yes, we’re about to discuss the Python Web Server Gateway Interface (WSGI) specification.

So, a quick note if you aren’t familiar with WSGI. WSGI is a specification that proposes rules that web frameworks and web servers should follow in order that you, the framework user, can swap out one WSGI server (like uWSGI) for another (Gunicorn) and expect your Python web application to continue to function.

Django and WSGI

And now, back to our journey into deeper knowledge!

So, what happens after your Django view returns a StreamingHttpResponse? In most Python web applications, the response is passed off to a WSGI server like uWSGI or Gunicorn (AKA, Green Unicorn).

As with HttpResponse, Django ensures that StreamingHttpResponse conforms to the WSGI spec, which states this:

When called by the server, the application object must return an iterable yielding zero or more bytestrings. This can be accomplished in a variety of ways, such as by returning a list of bytestrings, or by the application being a generator function that yields bytestrings, or by the application being a class whose instances are iterable.

Here’s how StreamingHttpResponse satisfies these requirements (full source):

    @property
    def streaming_content(self):
        return map(self.make_bytes, self._iterator)
# ...

    def __iter__(self):
        return self.streaming_content

You give the class a generator and it coerces the values that it produces into bytestrings.

Compare that with the approach in HttpResponse (full source):

    @content.setter
    def content(self, value):
        # ...
        self._container = [value]

    def __iter__(self):
        return iter(self._container)

Ah ha! An iterator with a single item. Very interesting. Now, let’s take a look at what a WSGI server does with these two different responses.

The WSGI server

Gunicorn’s synchronous worker offers a good example of what happens after Django returns a response object. The code is relatively short — here’s the important part (for our purposes):

respiter = self.wsgi(environ, resp.start_response)
try:
    if isinstance(respiter, environ['wsgi.file_wrapper']):
        resp.write_file(respiter)
    else:
        for item in respiter:
            resp.write(item)
    resp.close()

Whether your response is streaming or not, Gunicorn iterates over it and writes each string the response yields. If that’s the case, then what makes your streaming response actually “stream”?

First, some conditions must be true:

  • The client must be speaking HTTP/1.1 or newer
  • The request method wasn’t a HEAD
  • The response does not include a Content-Length header
  • The response status wasn’t 204 or 304

If these conditions are true, then Gunicorn will add a Transfer-Encoding: chunked header to the response, signaling to the client that the response will stream in chunks.

In fact, Gunicorn will respond with Transfer-Encoding: chunked even if you used an HttpResponse, if those conditions are true!

To really stream a response, that is, to send it to the client in pieces, the conditions must be true, and your response needs to be an iterable with multiple items.

What does the client get?

If the streaming response worked, the client should get an HTTP 1.1 response with the Transfer-Encoding: chunked header, and instead of a single piece of content with a Content-Length, the client should see each bytestring that your generator/iterator yielded, sent with the length of that chunk.

Here is an example that uses the code in this repository:

(streaming_django) ❯ curl -vv --raw "http://192.168.99.100/download_csv_streaming"
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 80 (#0)
> GET /download_csv_streaming HTTP/1.1
> Host: 192.168.99.100
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.11.1
< Date: Fri, 29 Jul 2016 14:27:58 GMT
< Content-Type: text/csv
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< Content-Disposition: attachment; filename=big.csv
<
f
One,Two,Three

f
Hello,world,1

...

10
Hello,world,99

0

* Connection #0 to host 192.168.99.100 left intact

So there you have it. We journeyed from considering when to use StreamingHttpResponse over HttpResponse, to an example of using the class in your Django project, then into the dungeons of WSGI and WSGI servers, and finally to the client’s experience. And we managed to stream a response — go us!

Refactoring search from Django app to microservice

One of my favorite technical book series is The Architecture of Open Source Applications. Here is how the site describes the goal of the series: In these two books, the authors of four dozen open source applications explain how their software is structured, and why. What are each program's major components? How do they interact? […]

Continue reading...

Dear Alma: How to quit drinking

Dear Alma, I assume that if you’re reading this letter for its true purpose, you could use some advice. Otherwise, file it away as a curiosity until the time comes — and I hope it never does. I have been sober now continuously since January 28, 2009, which at the time of this writing is […]

Continue reading...

Becoming a better learner

My post Becoming a better learner went up on the Safari blog recently. I really enjoyed digging into the latest brain research on my quest to become a better learner!

Continue reading...

Dear Alma: One day left to live

Dear Alma, Last night, I took a walk under the evening sky as it showed a color that seemed to drink me in, an aquamarine like crystal mined from some planet of the outer solar system, a true mystery of a color. I thought it was the color of life, not only green as I […]

Continue reading...

Protect against bias while hiring

I wrote a post recently on the Safari blog, this time about how to protect against bias while hiring. I often interview candidates at Safari, and I take that responsibility very seriously. My post describes actions that people can take at all stages of hiring to protect applicants from bias, both conscious and unconscious.

Continue reading...

How to thrive as a remote worker

I wrote a post this week about remote work, specifically how to avoid going crazy and losing your job when employed as such, on Safari’s corporate blog. Now that I’ve worked for more than three years as a fully remote developer, I have a lot of advice to share based on hard-won experience. Several posts […]

Continue reading...

PyCharm: Open the current file in Vim, Emacs or Sublime Text

Setting the program path for an external tool in PyCharm

Even though I use PyCharm, I still drop into Vim occasionally to edit configuration files. This was an annoying process until today, when I discovered that PyCharm and other Intellij editors can open the current file in an external tool. This works with both GUI and console-based applications, and its most trivial use case seems […]

Continue reading...

Video of my DjangoCon talk “The evolution of a RESTful Django backend”

The video of my first ever conference talk is now available on YouTube. Check it out here or watch it below. It could have been better, but least now my daughter will always be able to find a video of her dad looking foolish on the internet. The talk was a look at different web […]

Continue reading...

An Epic Review of PyCharm 3 from a Vim User’s Perspective

Code Completion

This review is for the Professional Edition of PyCharm 3. It includes screenshots and sound-free video demos of PyCharm features. I will try to cut straight to the point while offering some tips from my experience. My perspective is that of a professional software developer who has used Vim, Emacs, Sublime Text, PyDev and others. […]

Continue reading...