Summary

When the compiler generates a dynamic library, alter the default behavior to favor linking all dependencies statically rather than maximizing the number of dynamic libraries. This behavior can be disabled with the existing -C prefer-dynamic flag.

Motivation

Long ago rustc used to only be able to generate dynamic libraries and as a consequence all Rust libraries were distributed/used in a dynamic form. Over time the compiler learned to create static libraries (dubbed rlibs). With this ability the compiler had to grow the ability to choose between linking a library either statically or dynamically depending on the available formats available to the compiler.

Today's heuristics and algorithm are documented in the compiler, and the general idea is that as soon as "statically link all dependencies" fails then the compiler maximizes the number of dynamic dependencies. Today there is also not a method of instructing the compiler precisely what form intermediate libraries should be linked in the source code itself. The linkage can be "controlled" by passing --extern flags with only one per dependency where the desired format is passed.

While functional, these heuristics do not allow expressing an important use case of building a dynamic library as a final product (as opposed to an intermediate Rust library) while having all dependencies statically linked to the final dynamic library. This use case has been seen in the wild a number of times, and the current workaround is to generate a staticlib and then invoke the linker directly to convert that to a dylib (which relies on rustc generating PIC objects by default).

The purpose of this RFC is to remedy this use case while largely retaining the current abilities of the compiler today.

Detailed design

In english, the compiler will change its heuristics for when a dynamic library is being generated. When doing so, it will attempt to link all dependencies statically, and failing that, will continue to maximize the number of dynamic libraries which are linked in.

The compiler will also repurpose the -C prefer-dynamic flag to indicate that this behavior is not desired, and the compiler should maximize dynamic dependencies regardless.

In terms of code, the following patch will be applied to the compiler:

diff --git a/src/librustc/middle/dependency_format.rs b/src/librustc/middle/dependency_format.rs
index 8e2d4d0..dc248eb 100644
--- a/src/librustc/middle/dependency_format.rs
+++ b/src/librustc/middle/dependency_format.rs
@@ -123,6 +123,16 @@ fn calculate_type(sess: &session::Session,
             return Vec::new();
         }

+        // Generating a dylib without `-C prefer-dynamic` means that we're going
+        // to try to eagerly statically link all dependencies. This is normally
+        // done for end-product dylibs, not intermediate products.
+        config::CrateTypeDylib if !sess.opts.cg.prefer_dynamic => {
+            match attempt_static(sess) {
+                Some(v) => return v,
+                None => {}
+            }
+        }
+
         // Everything else falls through below
         config::CrateTypeExecutable | config::CrateTypeDylib => {},
     }

Drawbacks

None currently, but the next section of alternatives lists a few other methods of possibly achieving the same goal.

Alternatives

Disallow intermediate dynamic libraries

One possible solution to this problem is to completely disallow dynamic libraries as a possible intermediate format for rust libraries. This would solve the above problem in the sense that the compiler never has to make a choice. This would also additionally cut the distribution size in roughly half because only rlibs would be shipped, not dylibs.

Another point in favor of this approach is that the story for dynamic libraries in Rust (for Rust) is also somewhat lacking with today's compiler. The ABI of a library changes quite frequently for unrelated changes, and it is thus infeasible to expect to ship a dynamic Rust library to later be updated in-place without recompiling downstream consumers. By disallowing dynamic libraries as intermediate formats in Rust, it is made quite obvious that a Rust library cannot depend on another dynamic Rust library. This would be codifying the convention today of "statically link all Rust code" in the compiler itself.

The major downside of this approach is that it would then be impossible to write a plugin for Rust in Rust. For example compiler plugins would cease to work because the standard library would be statically linked to both the rustc executable as well as the plugin being loaded.

In the common case duplication of a library in the same process does not tend to have adverse side effects, but some of the more flavorful features tend to interact adversely with duplication such as:

Globals with significant addresses (statics). These globals would all be duplicated and have different addresses depending on what library you're talking to.
TLS/TLD. Any "thread local" or "task local" notion will be duplicated across each library in the process.

Today's design of the runtime in the standard library causes dynamically loaded plugins with a statically linked standard library to fail very quickly as soon as any runtime-related operations is performed. Note, however, that the runtime of the standard library will likely be phased out soon, but this RFC considers the cons listed above to be reasons to not take this course of action.

Allow fine-grained control of linkage

Another possible alternative is to allow fine-grained control in the compiler to explicitly specify how each library should be linked (as opposed to a blanked prefer dynamic or not).

Recent forays with native libraries in Cargo has led to the conclusion that hardcoding linkage into source code is often a hazard and a source of pain down the line. The ultimate decision of how a library is linked is often not up to the author, but rather the developer or builder of a library itself.

This leads to the conclusion that linkage control of this form should be controlled through the command line instead, which is essentially already possible today (via --extern). Cargo essentially does this, but the standard libraries are shipped in dylib/rlib formats, causing the pain points listed in the motivation.

As a result, this RFC does not recommend pursuing this alternative too far, but rather considers the alteration above to the compiler's heuristics to be satisfactory for now.

Unresolved questions

None yet!

6.7 KiB Raw Permalink Blame History