examples/langserver_ext.py processes are taking too much memory

[w3ichen]

Issue: subprocesses created from langserver_ext.py take up too much memory. Often, one process takes over 100MB. How can I reduce the memory size of each process?

I also found that the processes did not terminate when the websocket connection had closed. So the host was accumulating all of the processes until memory hit 100% and then the host crashes.

I fixed this issue by calling terminate on the subprocess inside the websocket's on_close

class PythonLanguageServer(websocket.WebSocketHandler):
    """Setup tornado websocket handler to host an external language server."""

    def open(self, *args, **kwargs):
        log.info(f"[{datetime.now().strftime('%d/%b/%Y %H:%M:%S')}] New Connection")
        print(f"[{datetime.now().strftime('%d/%b/%Y %H:%M:%S')}] New Connection")

        # Create an instance of the language server
        self.proc = process.Subprocess(["pyls"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

        # Create a writer that formats json messages with the correct LSP headers
        self.writer = streams.JsonRpcStreamWriter(self.proc.stdin)

        # Create a reader for consuming stdout of the language server. We need to
        # consume this in another thread
        def consume():
            # Start a tornado IOLoop for reading/writing to the process in this thread
            ioloop.IOLoop()
            reader = streams.JsonRpcStreamReader(self.proc.stdout)
            reader.listen(lambda msg: self.write_message(json.dumps(msg)))

        thread = threading.Thread(target=consume)
        thread.daemon = True
        thread.start()

    def on_message(self, message):
        """Forward client->server messages to the endpoint."""
        self.writer.write(json.loads(message))  # Most memory is used by this

    def check_origin(self, origin):
        return True

    def on_close(self):
        self.proc.stdin.close()
        self.proc.stdout.close()
        self.proc.proc.terminate()