Running vim as a server without X

So far, running vim as a server without X is impossible as far as I know. I don't understand why vim depends on X at all, especially for this case.

$ vim --servername foobarbaz
doesn't work, and even worse, silently fails. No warnings, errors, etc.

I thought it could be that vim didn't have some kind of compile option, but:
$ vim --version | grep client
+clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments

The only way to use vim's built in server is to give it a valid DISPLAY value at startup. This makes vim behave a little slower, which is especially noticeable at startup.

DISPLAY=:11 vim --servername foobarbaz

---

---

To start a server:
DISPLAY=:0.0 vim --servername mainVim

To edit a file on an already running vim server:
DISPLAY=:0.0 vim --servername MAINVIM --remote-tab-silent `find -type f -name filename`