Subject: integration of c-ares into various file descriptor based main loops [reposted]

integration of c-ares into various file descriptor based main loops [reposted]

From: Pavel Simerda <psimerda_at_redhat.com>
Date: Thu, 7 Nov 2013 13:14:56 -0500 (EST)

Hello,

apparently posting to c-ares mailing list is not possible for unsubscribed people. Therefore some you only received bits of my original e-mail in Jakub's reply. I'm reposting the original e-mail now subscrubed in hope we can somehow fix the communication gap. Original text follows...

****

Hi,

I started a new network name resolver library project (extending the idea of glibc's nsswitch) that is using c-ares as its backend. I had the chance to design both the external API and the backend API from scratch. While I was first searching for inspiration in the c-ares API, I realized that `ares_fds()` style API is suboptimal (unless the mainloop is indeed based on `select()`).

When designing the APIs, I considered the following well-known options:

 * select
 * poll
 * epoll
 * libevent
 * glib main loop

All of them has more or less native support for adding and removing file descriptors but for example *epoll* doesn't have support for inspection. My APIs take this into account and so they are based on *addition*, *modification* and *removal* of file descriptors from the multiplex, all of them handled by one function with the following arguments:

 * driver: the epoll fd, resolver struct pointer, etc...
 * fd: the file descriptor
 * events: bit array of monitored events, non-zero for *add*/*modify*, zero for *delete*

The main difference from `ares_fds()` is that for external API this has to be implemented using callbacks (or similar mechanism) bound directly to the places where the library needs to start or stop listening to file descriptor events. The supplied *callback* then handles addition/modification/removal of the file descriptors and you don't have to retrieve the whole file descriptor set before each `select()`, `poll()`, `epoll_wait()`, `g_main_context_iteration()` nor transform it into a data structure suitable for the respective main loop waiting method.

An *epoll* example can be found in the source code of the library:

  11 void
  12 watch_fd(netresolve_t resolver, int fd, int events, void *user_data)
  13 {
  14 int epoll_fd = *(int *) user_data;
  15
  16 struct epoll_event event = { .events = events, .data = { .fd = fd} };
  17
  18 if (epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, &event) == -1 && errno != ENOENT) {
  19 perror("epoll_ctl");
  20 abort();
  21 }
  22 if (events && epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &event) == -1) {
  23 perror("epoll_ctl");
  24 abort();
  25 }
  26 }

https://sourceware.org/git/?p=netresolve.git;a=blob;f=tests/test-async.c;h=fc146cd41a45ca744e382ab0258319c1a35d1881;hb=HEAD#l11

On the other hand, the added complexity of `ares_fds()` approach can be easily seen in the DNS backend:

  17 void
  18 register_fds(Data *data)
  19 {
  20 netresolve_backend_t resolver = data->resolver;
  21 fd_set rfds;
  22 fd_set wfds;
  23 int nfds, fd;
  24
  25 FD_ZERO(&rfds);
  26 FD_ZERO(&wfds);
  27 nfds = ares_fds(data->channel, &rfds, &wfds);
  28
  29 for (fd = 0; fd < nfds || fd < data->nfds; fd++) {
  30 if (!FD_ISSET(fd, &rfds) && FD_ISSET(fd, &data->rfds)) {
  31 FD_CLR(fd, &data->rfds);
  32 netresolve_backend_watch_fd(resolver, fd, 0);
  33 } else if (FD_ISSET(fd, &rfds) && !FD_ISSET(fd, &data->rfds)) {
  34 FD_SET(fd, &data->rfds);
  35 netresolve_backend_watch_fd(resolver, fd, POLLIN);
  36 }
  37 if (!FD_ISSET(fd, &wfds) && FD_ISSET(fd, &data->wfds)) {
  38 FD_CLR(fd, &data->wfds);
  39 netresolve_backend_watch_fd(resolver, fd, 0);
  40 } else if (FD_ISSET(fd, &wfds) && !FD_ISSET(fd, &data->wfds)) {
  41 FD_SET(fd, &data->wfds);
  42 netresolve_backend_watch_fd(resolver, fd, POLLOUT);
  43 }
  44 }
  45
  46 data->nfds = nfds;
  47
  48 if (!nfds)
  49 netresolve_backend_finished(resolver);
  50 }

https://sourceware.org/git/?p=netresolve.git;a=blob;f=backends/dns.c;h=7a006f497d0b24652285f9e3d5d55a463481c6ae;hb=HEAD#l17

Also `ares_process_fd()` seems to assume `select()`.

 110 void
 111 dispatch(netresolve_backend_t resolver, int fd, int events)
 112 {
 113 Data *data = netresolve_backend_get_data(resolver);
 114
 115 int rfd = events & POLLIN ? fd : ARES_SOCKET_BAD;
 116 int wfd = events & POLLOUT ? fd : ARES_SOCKET_BAD;
 117
 124 ares_process_fd(data->channel, rfd, wfd);
 125 register_fds(data);
 126 }

https://sourceware.org/git/?p=netresolve.git;a=blob;f=backends/dns.c;h=7a006f497d0b24652285f9e3d5d55a463481c6ae;hb=HEAD#l110

Would you consider an additional API as an alternative to `ares_fds()`, `ares_timeout()` and `ares_process_fd()` consisting of functions similar to the following ones useful?

 * void ares_set_fd_callback(ares_channel channel, int (*callback)(int fd, int events, void *user_data), void *user_data)
 * void ares_set_timeout_callbacks(ares_channel channel, int (*create)(int seconds, void *user_data), int (*remove)(int timeout_id), void *user_data)
 * int ares_dispatch_fd(ares_channel channel, int fd, int events)

Apart from possible improvements of *c-ares*, I will be happy for any feedback for the API I'm using in the library. It's in very early stage of development and needs more eyes than just the two of mine.

Cheers,

Pavel
Received on 2013-11-08