Neovim System Theme Using D-Bus

Introduction - The Idea

Although I typically prefer a dark color scheme across my system, especially in my text editor (Neovim btw), but some lighting situations demand a light color scheme for legibility.

Because of this, I set out to make switching color schemes on my system as simple as possible. Darkman gave me a good start. Simply put, it allows changing the system color scheme on the fly either manually, or on some automated schedule (sunrise/sunset for example). It does this primarily by acting as a system color scheme provider through D-Bus¹.

Two of my most used applications on my system are my web browser and my text editor. Firefox supports switching the D-Bus color scheme by default, but Neovim doesn’t, so I would have to add support myself.

The idea was simple. If I could interact with the system D-Bus from within Neovim, I would be able to switch Neovim’s color scheme on the fly whenever a D-Bus signal was emitted to notify us that the system theme changed.

Setting Up the Plugin

My first task at hand would be figuring out how to process D-Bus messages within Neovim.

As any lazy good developer would do, I checked to see if there was already a D-Bus library for Lua², and I quickly found dbus_proxy from LuaRocks. LuaRocks “allows you to create and install Lua modules as self-contained packages called rocks.” Essentially, LuaRocks is a package manager for Lua like Cargo is for Rust or npm is for Node.js.

Luckily for me, the lazy.nvim plugin manager recently added support for installing LuaRocks packages for Neovim plugins.

Specifying Plugin Dependencies

One of the ways lazy.nvim supports installing LuaRocks packages is through a Rockspec file. This file allows me to specify dependencies from LuaRocks, and lazy.nvim will automatically install them and make them available for my plugin to use directly.

I created the system_theme.nvim-scm-1.rockspec file in my plugin repo, and lazy.nvim automatically detected it and installed the necessary dbus_proxy dependency:

rockspec_format = "3.0"
package = "system_theme.nvim"
version = "scm-1"

dependencies = {
    "lua == 5.1",
    "dbus_proxy == 0.10.3",
}

source = {
    url = "git://github.com/cosmicboots/system-theme.nvim.git"
}

Sending D-Bus Requests

Following the documentation for dbus_proxy (and after fiddling around with the D-Bus parameters), I was able to construct a proxy and send a request for the current system theme:

local proxy = p.Proxy:new({
    bus = p.Bus.SESSION,
    name = "org.freedesktop.portal.Desktop",
    path = "/org/freedesktop/portal/desktop",
    interface = "org.freedesktop.portal.Settings",
})
local color_scheme = proxy:ReadOne("org.freedesktop.appearance", "color-scheme")
print(color_scheme) -- Prints 1 for dark 2 for light

This felt straight forward and I thought I was well on the way to getting my live theme swapping working… Turns out, there was much more in store for me.

Receiving D-Bus Signals (Part 1)

At this point, I tried to receive D-Bus signals so I would be alerted any time the system color scheme changed. Again, following the dbus_proxy documentation, I tried to attach a function to a signal:

proxy:connect_signal(function(pxy, ns, k, v)
    assert(pxy == proxy)
    if ns == "org.freedesktop.appearance" and k == "color-scheme" then
        print("color scheme changed to " .. v .. "!")
    end
end, "SettingChanged")

Because this was almost copied verbatim from the documentation, I expected something to happen, but I never got a message printed. After an embarrassingly long time, I rechecked the documentation and noticed a paragraph near the top that said:

For all this to work though, the code must run inside GLib’s main event loop. This can be achieved in two ways…

Turns out “all this” included listening for D-Bus signals. So without thinking much about it, I followed the documentation’s instructions and setup the GLib main loop below my signal code:

local GLib = require("lgi").GLib
-- Set up the application, then do:
local main_loop = GLib.MainLoop()
main_loop:run()

After I ran the code, Neovim froze!

The experienced reader might be able to guess exactly what just happened. Turns out, as the name suggests, main_loop:run() runs the main loop for GLib. The main loop is designed to run indefinitely to “manage all the available sources of events for GLib and GTK applications.” (source). This prevents Neovim’s main event loop from continuing until the GLib main loop is terminated.

Well, I obviously couldn’t just lock up Neovim’s event loop and expect my editor to still work, so I investigated the second option the dbus_proxy docs provided for running the GLib main loop:

Use more fine-grained control by running an iteration at a time from the main context; this is particularly useful when you want to integrate your code with an external main loop:

local GLib = require("lgi").GLib
-- Set up the code, then do
local ctx = GLib.MainLoop():get_context()
-- Run a single non-blocking iteration
if ctx:iteration() == true then
  print("something changed!")
end
-- Run a single blocking iteration
if ctx:iteration(true) == true then
  print("something changed here too!")
end

This section of the documentation suggested that I would be able to tie the GLib event loop to the Neovim event loop. I had hoped that I could use vim.schedule() to schedule the GLib loop iterations how I wanted; however, it turns out I wasn’t sure what the documentation for vim.schedule({fn}) meant when it said “Schedules {fn} to be invoked soon by the main event-loop.” Abusing schedule() would rapidly lock up Neovim.

At this point, I knew that I would have to architect the code in a more sophisticated way.

Multi-Threading

The problem as I understood so far was that I needed to run the GLib event loop, without locking up the execution of the main Neovim event loop. This sounded like a job for multi-threading, but I had no idea how to handle threads inside of Neovim, so I had to jump into how the Neovim event loop works.

Threads in Neovim Plugins

Neovim uses luv for it’s main event loop, and exposes this under the Lua API through vim.ui. Luv acts as a Lua wrapper around the battle-tested libuv library originally developed for Node.js.

The Neovim documentation gives a good starting place to multi-thread a plugin:

Plugins can perform work in separate (os-level) threads using the threading APIs in luv, for instance vim.uv.new_thread. Note that every thread gets its own separate Lua interpreter state, with no access to Lua globals in the main thread. Neither can the state of the editor (buffers, windows, etc) be directly accessed from threads.

In other words, I can run code in a separate thread so long as it doesn’t directly interact with the editor state.

Importing Modules In Threads

With this, I moved my GLib main loop into it’s own thread and tried to construct the proxy object:

local function thread_func()
    local p = require("dbus_proxy")
    local GLib = require("lgi").GLib

    local proxy = p.Proxy:new({
        bus = p.Bus.SESSION,
        name = "org.freedesktop.portal.Desktop",
        path = "/org/freedesktop/portal/desktop",
        interface = "org.freedesktop.portal.Settings",
    })

    local color_scheme = proxy:ReadOne("org.freedesktop.appearance", "color-scheme")
    print(color_scheme) -- Prints 1 for dark 2 for light

    local main_loop = GLib.MainLoop()
    main_loop:run()
end

local function run()
    local t = vim.uv.new_thread(thread_func)
end

Running this very quickly resulted in an error:

Error in luv thread:
[string ":source (no file)"]:2: module 'dbus_proxy' not found:
	no field package.preload['dbus_proxy']
	no file './dbus_proxy.lua'
	no file '/usr/share/luajit-2.1/dbus_proxy.lua'
	no file '/usr/local/share/lua/5.1/dbus_proxy.lua'
	no file '/usr/local/share/lua/5.1/dbus_proxy/init.lua'
	no file '/usr/share/lua/5.1/dbus_proxy.lua'
	no file '/usr/share/lua/5.1/dbus_proxy/init.lua'
	no file './dbus_proxy.so'
	no file '/usr/local/lib/lua/5.1/dbus_proxy.so'
	no file '/usr/lib/lua/5.1/dbus_proxy.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'

It seemed I’m not able to import the dbus_proxy module inside the thread. This struck me as strange, because the documentation seemed to suggest that I would have access to the same libraries inside my newly spawned thread:

A subset of the vim.* API is available in threads. This includes:

• …
• require in threads can use Lua packages from the global |package.path|
• …

For whatever reason, I found this to not be the case:

• package.path in main thread: ./?.lua;/usr/share/luajit-2.1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;/home/<user>/.local/share/nvim/lazy-rocks/system-theme.nvim/share/lua/5.1/?.lua;/home/<user>/.local/share/nvim/lazy-rocks/system-theme.nvim/share/lua/5.1/?/init.lua;;/home/<user>/.local/share/nvim/lazy-rocks/telescope.nvim/share/lua/5.1/?.lua;/home/<user>/.local/share/nvim/lazy-rocks/telescope.nvim/share/lua/5.1/?/init.lua;;/home/<user>/.local/share/nvim/lazy-rocks/unicode_picker.nvim/share/lua/5.1/?.lua;/home/<user>/.local/share/nvim/lazy-rocks/unicode_picker.nvim/share/lua/5.1/?/init.lua;
• package.path in child thread: ./?.lua;/usr/share/luajit-2.1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua

A simple fix (really a hack) is to pass in the parent thread’s package.path (and package.cpath) and manually update it inside the thread:

local function thread_func(pkg_path, pkg_cpath)
    package.path = pkg_path -- Update the child thread's package paths
    package.cpath = pkg_cpath
    local p = require("dbus_proxy")
    local GLib = require("lgi").GLib

    local proxy = p.Proxy:new({
        bus = p.Bus.SESSION,
        name = "org.freedesktop.portal.Desktop",
        path = "/org/freedesktop/portal/desktop",
        interface = "org.freedesktop.portal.Settings",
    })

    local color_scheme = proxy:ReadOne("org.freedesktop.appearance", "color-scheme")
    print(color_scheme) -- Prints 1 for dark 2 for light

    local main_loop = GLib.MainLoop()
    main_loop:run()
end

local function run()
    local t = vim.uv.new_thread(thread_func, package.path, package.cpath)
end

Now the libraries should import properly and we should see 1 or 2 printed out.

Receiving D-Bus Signals (Part 2)

With the second thread running the GLib main loop, we can try signals again:

local function thread_func(pkg_path, pkg_cpath)

    -- ... proxy setup omitted ...

    proxy:connect_signal(function(pxy, ns, k, v)
        assert(pxy == proxy)
        if ns == "org.freedesktop.appearance" and k == "color-scheme" then
            print("Theme changed to " .. v "!")
        end
    end, "SettingChanged")

    local main_loop = GLib.MainLoop()
    main_loop:run()
end

Now when the system theme changes, we can see the change printed out in real time.

Sending Inter-Thread Messages

Now that threads seem to be working, we need to be able to pass messages between them. Sadly I couldn’t find any Go or Rust style “channels” in the Neovim documentation for inter-thread communication. I did however, find pipes, which would allow sending messages across threads just the same, albeit with a more primitive UNIX style API.

Using a pipe, we can listen for a D-Bus signal and send updates across the wire using pipe:write() and pipe:read_start():

local function thread_func(pkg_path, pkg_cpath, pipe_fd)

    -- ... proxy setup omitted ...

    local pipe = vim.uv.new_pipe()
    pipe:open(pipe_fd)

    proxy:connect_signal(function(pxy, ns, k, v)
        assert(pxy == proxy)
        if ns == "org.freedesktop.appearance" and k == "color-scheme" then
            pipe:write(v)
        end
    end, "SettingChanged")

    local main_loop = GLib.MainLoop()
    main_loop:run()
end

local function run()
    local fds = vim.uv.pipe()
    assert(fds ~= nil)

    local pipe = vim.uv.new_pipe()
    pipe:open(fds.read)

    pipe:read_start(function(err, data)
      assert(not err, err)
      print("Color scheme changed to" .. data)
    end)

    local t = vim.uv.new_thread(thread_func, fds.write)
end

Cleaning Up The Thread

At this point, most the basic logic is setup to perform theme switching on D-Bus events: just add logic to the pipe:read_start() callback function inside run().

Any good systems engineer will notice that we never clean up or stop our thread. Because libuv doesn’t provide a way to kill threads³, we have to have a way to signal our thread to stop.

If we create a new pipe, we can send a message to the thread to signal it to stop, similar to how we’re sending the theme from the thread right now. We just need to make sure we pass in the file descriptor (fd) for the read side of the pipe:

local function thread_func(pkg_path, pkg_cpath, pipe_fd, close_fd)

    -- ... proxy and signal setup omitted ...

    local close_pipe = vim.uv.new_pipe()
    close_pipe:open(close_fd)

    local main_loop = GLib.MainLoop()

    -- Notice we setup reading from the pipe /after/ creating the main loop,
    -- but /before/ we run it
    close_pipe:read_start(function(err, data)
        assert(not err, err)
        main_loop:quit() -- kill the main loop allowing the thread to terminate
    end)

    main_loop:run()
end

local function run()

    --- ... other pipe code omitted ...

    local close_fds = vim.uv.pipe()
    assert(close_fds ~= nil)

    local t = vim.uv.new_thread(thread_func, fds.write, close_fds.read)
end

Now, as soon as we write any data into into the close_fds.write end of the pipe, the thread should terminate. At least, that’s what we would expect.

Battle with GLib Main Loop

Unfortunately, we’ve run into our nemesis again: the GLib main loop.

This little section of code is waiting for data to be available in the pipe:

close_pipe:read_start(function(err, data)
    assert(not err, err)
    main_loop:quit() -- kill the main loop allowing the thread to terminate
end)

What isn’t obvious from this code is that we’re actually calling into the libuv event loop bundled with Neovim. The code isn’t waiting on data in the pipe, but rather telling Neovim’s event loop to schedule our callback function whenever it detects data in the pipe.

What actually runs the callback function? The Neovim event loop!

Neovim isn’t able to run our callback because the GLib main loop is preventing the Neovim event loop from running.

Well then, what’s the solution? How can we read from the pipe if the GLib main loop is hogging resources? The answer is with GLib.

Before we solve this, we need to establish two facts:

1. We’re working with two event loops: the libuv loop and the GLib loop. Both of these event loops accomplish similar things. Think of them as our “standard library” to access threads, the file system, and other IO.
2. The “pipe” we constructed with Neovim’s vim.uv.new_pipe() function is really just a pair of read and write file descriptors.

Knowing both these facts we might wonder: “what if we can read from the pipe using GLib rather than Neovim’s API?”

That’s exactly what I asked myself.

The Lua bindings used for GLib are from lgi, which has incredibly sparse documentation, so I went directly to the GLib docs.

After some digging, I found the documentation for opening an IOChannel, which allows us open the file descriptor (close_fd) we passed into our child function.

Just opening the file descriptor isn’t enough; we need to register it with the GLib main loop so GLib knows when data is there. After some more digging, I found io_add_watch(), which allows us to do just that. We can register a callback function with io_add_watch() that runs when there’s data available in the pipe:

local function thread_func(pkg_path, pkg_cpath, pipe_fd, close_fd)

    -- ... proxy and signal setup omitted ...

    local main_loop = GLib.MainLoop()

    -- Notice we setup reading from the pipe /after/ creating the main loop,
    -- but /before/ we run it
    local io_chan = GLib.IOChannel.unix_new(close_fd)
    GLib.io_add_watch(io_chan, GLib.PRIORITY_DEFAULT, "IN", function()
        main_loop:quit() -- kill the main loop allowing the thread to terminate
    end)

    main_loop:run()
end

Notice we’re not using any calls to Neovim’s event loop (vim.uv.*) to read the close pipe file descriptor. Because reading the file descriptor is handled by the GLib main loop, we will still be able to receive incoming data while the GLib main loop is running.

Conclusion

Let’s bring everything back together to summarize the main issue with handling D-Bus messages inside a Neovim plugin.

D-Bus is notoriously hard to handle without language bindings. The official D-Bus low-level C API docs even states:

If you use this low-level API directly, you’re signing up for some pain.

That being said, language bindings are essentially required, and the best library for Lua ultimately wraps GLib, which requires it’s own loop to handle D-Bus signal events.

Because we can’t run two event loops at the same time, in the same thread, we have to multi-thread the system.

In the end, we ended up with two threads. The main thread handles interacting directly with Neovim (changing themes for example) and the child/worker thread handles the D-Bus requests. This approach works for what I’ve needed so far, and I feel confident that it would work for any other Neovim plugins that need to handle system D-Bus messages.

The complete code for this Neovim plugin can be found on GitHub.