OpenStack servers are designed with an integral web front-end – a pythonic wsgi mini “Web Server” which opens up its own socket and serves http requests directly. The incoming requests accepted by the integral web front-end are then forwarded to a wsgi application (the request processor) for further handling, possibly via wsgi middleware sub-components. See [A] in the diagram below.
In view of the limited support offered by the integral web front-end, some installations may also take advantage of a full-fledged Proxy Server in front of the installation (See [B] above).
An alternative design would be to strip the integral web front-end off the OpenStack code and use standard full-fledged Web Server instead. Under such an alternative design, OpenStack servers can be reincarnated as a set of request processors, extending any standard Web Server that includes wsgi support (e.g. Apache2) as shown in [C] above.
This post aims to collect the pros and cons of the current design – or else help answer the question: will OpenStack benefit from adding support for working as a request processor, attached to a standard WSGI server? The discussion examples will mainly concern Swift, as this is becoming a main focus for me these days, but can be generalized for others OpenStack packages.
Adam Young had started this thread of though as related to Keystone. Joshua Harlow responded by suggesting that: “…it should be possible for all WS endpoints to be hosted in apache (or other server)… This might be connected to extracting/abstractig out eventlet…“. This is inline with the subject of this post. Other good people views were collected and summarized below.
The OpenStack integral Web Front-End
The integral web front-end used by OpenStack is the eventlet wsgi one. Eventlet offers ‘standard’ threading interface to the python programmer while being implemented using asynchronous I/O underneath. See here a post discussing whether Asynchronous I/O makes sense in OpenStack.
A “Web Server” per Request Processor
Each OpenStack server has its own integral web front-end, each working on a single port. For example, an integrated Swift node may have four web front-ends: one for the Swift Proxy Server, one for the Swift Account Server, one at the Swift Container Server and one at the Swift Object Server. Each working on a dedicated port. One may decide to have these servers located on separate nodes (e.g Swift Proxy Server on an Interface Node and a Storage Node running the remaining servers, each with its integral Web Server).
A Web Server per Node
It is important to note that this design of plural web front-ends is a design choice and may be replaced by a single standard full-fledged Web Server (e.g. apache2). In such a case, Swift code will be stripped off the eventlet wsgi web front-end and will run as request processors off the full-fledged Web Server (e.g. using the wsgi_mod). A single Web Server for any request processor running on the Node. If anyone decides to run non-Swift OpenStack request processor (e.g. Keystone) or any non-OpenStack application, he/she may re-use the same Web Server for all its needs. There is nothing new in the alternative design presented here – it is a standard way of building web applications. Hence a comparison to the design choice in OpenStack is in place.
One outcome of this alternative design is that it becomes up to the server if Asynchronous I/O is used or not. Apache2 would use a combination of multiprocessing and multithreading approach but no Asynchronous I/O. As discussed in the post about Asynchronous I/O, some OpenStack request processors may benefit from Asynchronous I/O while others may best avoid it.
Note that when using a server like Apache2, which uses CPython threads, if multiple threads are needed to process a request, Eventlet’s green threads can still be used. In this case Asynchronous I/O will be used within the scope of a single request (no cooperation between threads of different requests).
Separation of Concerns
Each OpenStack server includes a dedicated request processor with a given function. Coupling the request processor function with that of a web front end forces the design to become a compromise. Take for example the concurrency level required by each server:
- The web front end needs to be highly concurrent such that any incoming TCP connection is answered before the underlying socket queue becomes full. While the web front end maintains all open client sessions (each as a thread), many of these sessions could be idle. After a response is sent back to the client, it is custom to keep the TCP session open, allowing the client to send its next request(1).
- The request processor on the other end, is session-less (as it is REST based) and should not maintain data with regard to any request that was completed. This is important to help scale the request processor. Further, if the request processor is I/O bound, it is important to avoid trashing it with more requests than it can process (See in this post more details). An adequate design would therefore be to queue the remaining requests rather than processing them concurrently. As shown in the next diagram, to help the request processor scale, it should concurrently process only a subset of the outstanding requests which is a subset of the open TCP sessions.
Today OpenStack servers put everything into one big asynchronous I/O pool. This means that the web front end which is coupled in the same process with the request processor share the same asynchronous I/O domain. As a result the two functions share the same concurrency level and there are no measures to ensure that each one has the right level of concurrency for best performance. The problem is even more severe if a server reaches a CPU bound.
Q: How can one ensure that while the web front end runs as many concurrent requests as it can, the concurrency level of the request processor is controlled? Q: Maybe add a queue to OpenStack’s common/wsgi.py implementation.
Which is a better design overall?
This 1M$Q depends on many factors including performance, manageability, security, scalability and other customer requirements. Beyond technical aspects there may be other considerations (e.g. enabling a customer to gain better control of the service). One of the reasons for putting out this post is to trigger people to consider the implications of the two designs.
Which is a better design technically?
Different factors may come to play here. The current design seem to be best tuned for memory bound servers that suffice with running on a single core and seek to service as many requests as possible concurrently. I/O bound applications may resort to trashing when using the current approach and could potentially scale better with the presented alternative approach. The patterns used by each application server may differ significantly. Hence, the right level of concurrency required should be tuned per server, in order to gain 100% utilization of each of the servers. Measures need to be designed in to allow such tuning and control.
One significant concern is that there seem to be no known measures to determine the scalability or resulting performance of the two designs. Further, a search in scholar for “greenlet eventlet performance” yields zero results. Eric Day posted results that includes an Eventlet micro benchmark Yet it is unclear how these results can be compared to the alternative node design discussed here. The man behind mod_wsgi, Graham Dumpleton, views that one can’t meaningfully compare two web servers detached from the specific application requirements. Free translation to our case could be – we need to support alternative wsgi servers as part of OpenStack such that people could evaluate which is the right/best server for their specific app. – {TBD ask Graham if this aligns with his view}.
OpenStack’s extendability
Adam Young raised the issue of using a standard Web Server instead of the integral web front-end for OpenStack’s Keystone. In his note he brought up an interesting argument. Eventlet’s Asynchronous I/O design is completely dependent on removing any blocking code from the application. As long as the code is written in Python, Eventlet has a trickery to monkey patch the code. This is likely to become a limiting factor for the usage of DB drivers written in C and used via a python interface. In fact any C extension code may be blocking, such that OpenStack’s extendability may become an issue.
Using a Proxy Server
In view of the limited feature support offered by the current OpenStack integral web front-end, it was suggested that administrators may deploy a Proxy Server in front of the OpenStack service. Yet it is unclear if such a Proxy would still be needed/used if a per-node full-fledged Web Server is used instead of the current per-server web front-end.
The next steps…
- This post is intended to be updated with any new information gathered on the subject.
- Adam Young posted a description how to use Keystone with httpd. I will be posting a description of how to use Swift with apache2.
- As I am working on attaching Swift as request processor to apache2, I am encountering some non standard behavior of the Swift wsgi application – these will hopefully be pushed back as bug fixes.
- Some comparative performance measurements are needed to gain understanding of specific examples.
- Push for blueprint to add such support: Swift first (and if the commonality is reasonable to other servers).
—————————————————————————————–
(1) thanks to Nadav Harel for pointing this up
David Hadas 