Reflecting On the Future of AppBundler

Deployment of GUI applications has always been a hassle for the developers. Challenges like development complexity for cross-platform targets, distribution, user adoption barriers, and maintainability have reached the point where web page has become a de facto distribution format for any new application development. However, web applications are often not an optimal solution when performance, hardware interfacing, offline functionality and custom visualisations are of concern, which describes requirements for many scientific/industrial software.

In recent times, Julia programming language has attained quite a foothold in this area with an unparalleled numerical software ecosystem from plain differential equations and quantum simulations to advanced visualisation frameworks that Makie can offer. GUI frameworks such as Gtk, QML, ImGUI, and Mousetrap are available along with evolving ecosystems around Pluto notebooks and Geary dashboards. However, deploying such applications to users who don't know or don't want to know Julia's workings significantly hinders its adoption for such scenarios.

Over the recent years, Julia has attained an unparalleled Artifacts system, allowing the developer to use already compiled binary artefacts via a system known as BinaryBuilder in Julia, which is tightly integrated into Julia's' package manager. This, for instance, enables easy distribution of project files and instantiating all relevant dependencies via Manifest.toml irrelevant to the operating system used, makes projects written in Julia easy to maintain long-term. As a result GUI applications can be developed on one platform and deployed confidently on another with no modifications whatsoever. This is possible as our favourite GUI frameworks are compiled and distributed across all major desktop operating systems, such as macOS, Linux, Windows, and even FreeBSD, using the BinaryBuilder infrastructure.

The recent push from OS vendors to run applications in sandboxed mode has made GUI applications more secure than ever, which can greatly instil confidence in trying out new software. On Linux, it is snap and flatpack; on Windows, it is MSIX format; and on macOS, it's the entitlement system. The sandboxed mode prevents the applications from gaining access to the parts of the system that the user had not allowed to and prevents the spread of malware. Much like today, we don't fear about opening webpages on the internet as the Javascript runtime is heavily sandboxed within the browser. However, deploying software into these formats is very challenging as Julia application developers often are not accustomed to thinking about dependency linking and intricacies/limitations within each sandboxed environment and the knowledge how to do so is advancing.

The AppBunlder project addresses these latter challenges by vendoring package dependencies and artifacts and offering recipes to cast them into mainstream desktop application installer bundles. The developer only needs to create an entry point and a project Manifest.toml file, which will retrieve the same dependencies for all operating systems on which the desktop application is developed. This makes it quick, predictable, and convenient for rapid iteration, which brings down the barriers to what a single developer can maintain. This was demonstrated for all major operating systems in my JuliaCon 2024 AppBundler talk. Furthermore, it is not limited to desktop applications. It can be used to deploy a server application on Linux via snap format or can also be used in Julia distributions with installed and precompiled dependencies where TTFX for large packages is a pain point in educational settings.

The AppBundler story is not yet finished, with several challenges and opportunities on the horizon. Developers need to maintain bundling postprocessing toolchains for MacOS and Windows. On Windows, the toolchain setup is particularly nasty as it requires installing the whole SDK and then pinpointing the correct executable in a deeply nested folder structure. As those tools are not open source, they can't be distributed with BinaryBuilder, and hence AppBundler.build, a function that produces finalised bundles on respective host systems, puts a big burden in the long term as it would encode many assumptions. This also makes it burdensome to support CI workflows for testing and deployment.

Open-source alternatives for MacOS have recently appeared, whereas Windows offers MSIX packaging software under an open-source license, albeit with limited documentation. If these open-source tools could be used instead of the proprietary alternatives, then the maintenance of AppBundler.build, a function that produces finalised bundles on the respective host systems, would become simple. Much like now, snap bundles can be created with mksquashfs, which is retrieved from Julia. This integration would facilitate the creation of GitHub action scripts for package releases upon codebase tagging and integration tests for early bug detection.

Furthermore, successfully integrating these open-source tools could also enable the creation of finalised bundles on Linux for all desktop platforms. The missing piece would be cross-compilation support for Julia, which is anticipated in the future. This development would significantly ease the deployment of GUI applications across platforms. In the meantime, a prototype could be implemented where the application is precompiled during the first run. That would provide another use case for cross-compilation and could work as a catalyst for the cross-compilation support efforts.

Generally, Julia's compilation capabilities have improved over time. The AppBundler uses PkgImages, which became relocatable only with Julia 1.11. Previously, the only compilation option was via SysImages, which continues to be exposed via PackageCompiler. PackageCompiler is slower to compile than PkgImages but can be essential to reduce startup times. Looking ahead to the future, we see that Julia will get the ability to be statically compiled via the Juliac project, which has shown promise in reducing binary sizes and being competitive with other statically compiled programming languages. All these options could be incorporated with the AppBundler, allowing a seamless switch between None|PkgImages|SysImages|JuliaC as compilation methods by passing a single structured keyword argument.

One of the hard parts about GUI applications is their distribution. For applications to get accepted in Snap, MacOS or Windows marketplaces, they must use the least privileges to get favourable reviews. Using existing marketplaces can also promote discovery, which is important for new open-source projects.

During few days after the JuliaCon hackathon, I managed to get MacOS sandboxing to work by furiously reproducing the sandboxing environment and found a single necessary entitlement:

<key>com.apple.security.temporary-exception.mach-lookup.global-name</key>
    <array>
    <string>com.apple.coreservices.launchservicesd</string>
</array>

which was a hell of a lot of work to discover, even when using ChatGPT. This only happened to GUI applications and seems to have been absent for the game engine Nathan Daly used when submitting a game to the MacOS app store bundled and built with ApplicationBuilder presented on JuliaCon 2018. One may inspect further why GUI frameworks like QML or Gtk need this and if it could be avoided.

Similar work needs to resume using Snap format and MSIX in sandboxed environments. I managed to get GLFW to work, but QML failed to render it, so I opened a bug report. Similar issues appeared with Gtk when loaded with OpenGL. The issue could be somewhere deep in how OpenGL is linked to QML when compiled, but who knows? It is going to be a bear to debug. Flatpack does not have such issues in the confined environment or has the same problem, which would be worth exploring. If the former is the case, I could write that this works on flatpack and hence snap ... prove me wrong at the snap discourse forum. However, I would need to make a flatpack-builder BinaryBuilder recipe to build flatpacks, which likely also involves compiling several of its dependencies.

Resolving sandboxing issues is particularly important for MSIX bundles, as the installation experience when they are self-signed is horrid compared to MacOS or Linux. From what I have gathered, Windows certificates cost around 100...200 euros/year. Hence, distribution in the Windows app store can be a free alternative, but they should not use fullTrust privileges for favourable reviews. Here, the issue starts with Julia failing to launch, as reported in issue #52007. There seems to be a clear path forward with explicitly making a launcher to load every DLL dependency. However, there is a possibility that the next set of difficulties would follow when starting GUI applications. In particular, Gtk applications don't run in the fullTrust environment, where the issue seems to be about assumptions on linking. This seems can be resolved easily, but it still needs to be done.

During JuliaCon, Michael Ferbst raised an interesting issue: The DFTK library, being so large, takes significant time to compile when instantiated on students' computers. This can also be a good use case for AppBundler, as it can precompile Julia's environment and expose the terminal as regular Julia does, which I have used extensively in debugging recipes. However, exposing this use case with API and documentation for creating such Julia distributions is essential. A logical extension would be to build Julia from the source and bundle it, but juliaup works well already.

There are many opportunities to enhance the AppBundler's user experience and make it ideal for distributing GUI and CMD applications. Overcoming these challenges would simplify AppBundler's maintenance, allowing for the creation of CI integration tests. In addition, it would result in a more user-friendly distribution experience for Julia applications, comparable to that of electron application distribution. This improvement would reduce the barriers to scientific software development in Julia, which has traditionally relied on less modern and user-friendly technologies like C++/Python. Additionally, it could expand Julia's ecosystem into nontraditional domains like cryptographic applications due to Julia's strong type safety and ease of use or games.

See also my JuliaCon 2024 talk on AppBundler.jl and follow new announcements and discussions on discourse.

You can also help to review my NLNET application on hackmd.io!