Limitations of Apache APISIX’s HTTP-Based etcd Operations

When etcd was in version 2.x, the API interface it exposed was HTTP 1 (we will refer to it as HTTP from now on). After the etcd was upgraded to version 3.x, it switched the protocol from HTTP to gRPC. For users that don’t support gRPC, etcd provides gRPC-Gateway to proxy HTTP requests as gRPC to access the new gRPC APIs.

When APISIX started using etcd, it used the etcd v2 API. In APISIX 2.0(2020), we have updated the requirement for etcd from version 2.x to 3.x. etcd’s compatibility with HTTP has saved us effort for the version update. We just needed to modify the code on calling methods and processing responses. However, throughout the years, we have also found some problems related to etcd’s HTTP API. There are still some subtle differences. We realized that having a gRPC-gateway does not mean it can perfectly support HTTP access.

Here’s a list of related issues we’ve encountered with etcd over the past few years:

  1. gRPC-gateway disabled by default. Due to the negligence of the maintainer, the default configuration of etcd does not enable gRPC-gateway in some projects. So we had to add instructions in the document to check whether the current etcd has gRPC-gateway enabled. See https://github.com/apache/apisix/pull/2940.
  2. By default, gRPC limits responses to 4MB. etcd removes this restriction in the SDK it provides but not in gRPC-gateway. It turns out that the official etcdctl (built on the SDK it provides) works fine, but APISIX doesn’t. See https://github.com/etcd-io/etcd/issues/12576.
  3. Same problem – this time with the maximum number of requests for the same connection. Go’s HTTP2 implementation has a MaxConcurrentStreams configuration that controls the number of requests a single client can send simultaneously, defaulting to 250. Which client would normally send more than 250 requests at the same time? So etcd has always used this configuration. However, gRPC-gateway, the “client” that proxies all HTTP requests to the local gRPC interface may exceed this limit. See https://github.com/etcd-io/etcd/issues/14185.
  4. After etcd enables mTLS, etcd uses the same certificate as both the server certificate and the client certificate: the server certificate for gRPC-gateway and the client certificate when gRPC-gateway accesses the gRPC interface. If the server auth extension is enabled on the certificate, but the client auth extension is not enabled, an error will result in certificate verification. Once again, accessing directly with etcdctl works fine(as the certificate will not be used as a client certificate in this case), but APISIX doesn’t. See https://github.com/etcd-io/etcd/issues/9785.
  5. After enabling mTLS, etcd allows security policies configuration of certificates’ user information. As mentioned above, gRPC-gateway uses a fixed client certificate when accessing the gRPC interface rather than the certificate information used to access the HTTP interface at the beginning. Thus, this feature will not work naturally since the client certificate is fixed and will not be changed. See https://github.com/apache/apisix/issues/5608.

We can summarize the problems in two points:

  1. gRPC-gateway (and perhaps other attempts to convert HTTP to gRPC) is not a silver bullet that fixes all problems.
  2. The developers of etcd don’t put enough emphasis on the HTTP to gRPC method. An