Add initial scaffolding for the GN build.

See "GN build roundtable summary; adding GN build files to the repo" on
llvm-dev and cfe-dev for discussion.

In particular, this build is completely unsupported. People adding new files to
LLVM are not expected to update the GN build files, and reviewers are not
supposed to request the gn build files to be updated.

This adds just enough to be able to build llvm/lib/Demangle. It requires using
a monorepo.

This adds a few build config options you can set in args.gn
(`gn args out/foo --list` for all):
- is_debug = true to enable debug builds (defaults to release)
- llvm_enable_assertions to toggle assertions (defaults to true)
- clang_base_path, if set an absolute path to a locally-built clang to be used
  as host compiler

Differential Revision: https://reviews.llvm.org/D54345

llvm-svn: 347128

llvm/utils/gn/.gn

@@ -0,0 +1,16 @@
+# FIXME: Once it's possible to add files to the root directory of the
+# monorepo, move this file to there.  Until then, you need to pass
+# `--dotfile=llvm/utils/gn/.gn --root=.` to the `gn gen` command.
+
+buildconfig = "//llvm/utils/gn/build/BUILDCONFIG.gn"
+
+# Disallow all calls to exec_script. We should be very conservative about
+# whitelisting things here.
+exec_script_whitelist = []
+
+# The normal GN setup is to have BUILD.gn files in-tree and a root BUILD.gn
+# file.  Since LLVM's GN build is unofficial, set secondary_source to tell GN
+# that e.g. llvm/lib/Demangle/BUILD.gn is found at
+# llvm/utils/gn/secondary/llvm/lib/Demangle/BUILD.gn and that the root BUILD.gn
+# file is at llvm/utils/gn/secondary/BUILD.gn
+secondary_source = "//llvm/utils/gn/secondary/"

llvm/utils/gn/README.rst

@@ -0,0 +1,129 @@
+=====================
+Building LLVM with GN
+=====================
+
+.. contents::
+   :local:
+
+.. _Introduction:
+
+Introduction
+============
+
+*Warning* The GN build is experimental and best-effort. It might not work,
+and if you use it you're expected to feel comfortable to unbreak it if
+necessary. LLVM's official build system is CMake, if in doubt use that.
+If you add files, you're expected to update the CMake build but you don't need
+to update GN build files. Reviewers should not ask authors to update GN build
+files. Keeping the GN build files up-to-date is on the people who use the GN
+build.
+
+*Another Warning* Right now, we're in the process of getting the GN build
+checked in. As of this writing, it's not yet functional at all. Check back
+in a few weeks!
+
+`GN <https://gn.googlesource.com/gn/>`_ is another metabuild system. It always
+creates ninja files, but it can create some IDE projects (MSVC, Xcode, ...)
+which then shell out to ninja for the actual build.
+
+Its main features are that GN is very fast (it currently produces ninja files
+for LLVM's build in 35ms on the author's laptop, compared to 66s for CMake) --
+a 2000x difference), and since it's so fast it doesn't aggressively cache,
+making it possible to switch e.g. between release and debug builds in one build
+directory.
+
+It is arguable easier to configure than the CMake build, and has native support
+for building with multiple toolchains in one build directory. The build
+description is declarative-ish, allowing GN to print it in a json format that
+can fairly easily be converted to other metabuild system inputs.
+
+The main motivation behind the GN build is that some people find it more
+convenient for day-to-day hacking on LLVM than CMake. Distribution, building
+just parts of LLVM, and embedding the LLVM GN build from other builds are a
+non-goal for the GN build.
+
+This is a `good overview of GN <https://docs.google.com/presentation/d/15Zwb53JcncHfEwHpnG_PoIbbzQ3GQi_cpujYwbpcbZo/edit#slide=id.g119d702868_0_12>`_.
+
+.. _Quick start:
+
+Quick start
+===========
+
+*Warning* Right now, we're in the process of getting the GN build checked in.
+As of this writing, it's not yet functional at all.
+
+GN only works in the monorepo layout.
+
+#. Obtain a `gn binary <https://gn.googlesource.com/gn/#getting-started>`_.
+
+#. In the root of the monorepo, run
+   `gn gen --dotfile=$PWD/llvm/utils/gn/.gn --root=. out/gn` (`out/gn` is the
+   build directory, it can have any name, and you can have as many as you want,
+   each with different build settings).
+
+#. Run e.g. `ninja -C out/gn check-lld` to build all prerequisites for and
+   run the LLD tests.
+
+By default, you get a release build with assertions enabled that targets
+the host arch. You can set various build options by editing `out/gn/args.gn`,
+for example putting `is_debug = true` in there gives you a debug build. Run
+`gn args --list out/gn` to see a list of all possible options. After touching
+`out/gn/args.gn`, just run ninja, it will re-invoke gn before starting the
+build.
+
+GN has extensive built-in help; try e.g. `gn help gen` to see the help
+for the `gen` command. The full GN reference is also `available online
+<https://gn.googlesource.com/gn/+/master/docs/reference.md>`_.
+
+GN has an autoformatter: `git ls-files '*.gn' '*.gni' | xargs -n 1 gn format`
+after making GN build changes is your friend.
+
+To not put `BUILD.gn` into the main tree, they are all below `utils/gn/tree`.
+For example, the build file for `llvm/lib/Support` is in
+`utils/gn/tree/llvm/lib/Support`.
+
+.. _Philosophy:
+
+Philosophy
+==========
+
+GN believes in using GN arguments to configure the build explicitly, instead
+of implicitly figuring out what to do based on what's available on the current
+system.
+
+configure is used for three classes of feature checks:
+
+- compiler checks. In GN, these could use exec_script to identify the host
+  compiler at GN time. For now the build has explicit toggles for compiler
+  features. (Maybe there could be a script that writes args.gn based on the
+  host compiler).  It's possible we'll use exec_script() for this going forward,
+  but we'd have one exec_script call to identify compiler id and version,
+  and then base GN arg default values of compiler id and version instead of
+  doing one exec_script per feature check.
+  (In theory, the config approach means a new os / compiler just needs to tweak
+  the checks and not the code, but in practice a) new os's / compilers are rare
+  b) they will require code changes anyhow, so the configure tradeoff seems
+  not worth it.)
+
+- library checks. For e.g. like zlib, GN thinks it's better to say "we require
+  zlib, else we error at build time" than silently omitting features. People
+  who really don't want to install zlib can explicitly set the GN arg to turn
+  off zlib.
+
+- header checks (does system header X exist). These are generally not needed
+  (just keying this off the host OS works fine), but if they should become
+  necessary in the future, they should be done at build time and the few
+  targets that need to know if header X exists then depend on that build-time
+  check while everything else can build parallel with it.
+
+- LLVM-specific build toggles (assertions on/off, debug on/off, targets to
+  build, ...). These map cleanly to GN args (which then get copied into
+  config.h in a build step).
+
+For the last two points, it would be nice if LLVM didn't have a single
+`config.h` header, but one header per toggle. That way, when e.g.
+`llvm_enable_terminfo` is toggled, only the 3 files caring about that setting
+would need to be rebuilt, instead of everything including `config.h`.
+
+GN doesn't believe in users setting arbitrary cflags from an environment
+variable, it wants the build to be controlled by .gn files.

llvm/utils/gn/build/BUILD.gn

@@ -0,0 +1,153 @@
+import("//llvm/utils/gn/build/buildflags.gni")
+import("//llvm/utils/gn/build/mac_sdk.gni")
+import("//llvm/utils/gn/build/toolchain/compiler.gni")
+import("//llvm/utils/gn/build/enable_threads.gni")
+
+config("compiler_defaults") {
+  # FIXME: Don't define this globally here.
+  if (host_os != "win") {
+    defines = [ "LLVM_ON_UNIX" ]
+  }
+
+  if (!llvm_enable_assertions) {
+    defines += [ "NDEBUG" ]
+  }
+
+  cflags = []
+
+  if (host_os == "mac" && clang_base_path != "") {
+    cflags += [
+      "-isysroot",
+      mac_sdk_path,
+    ]
+  }
+
+  if (host_os != "win") {
+    if (is_debug) {
+      cflags += [ "-g" ]
+    } else {
+      cflags += [ "-O3" ]
+    }
+    cflags += [ "-fdiagnostics-color" ]
+    cflags_cc = [
+      "-std=c++11",
+      "-fno-exceptions",
+      "-fno-rtti",
+      "-fvisibility-inlines-hidden",
+    ]
+  } else {
+    if (is_debug) {
+      cflags += [ "/Zi" ]
+    } else {
+      cflags += [
+        "/O2",
+        "/Zc:inline",
+      ]
+    }
+    defines += [
+      "_CRT_SECURE_NO_DEPRECATE",
+      "_CRT_SECURE_NO_WARNINGS",
+      "_CRT_NONSTDC_NO_DEPRECATE",
+      "_CRT_NONSTDC_NO_WARNINGS",
+      "_SCL_SECURE_NO_DEPRECATE",
+      "_SCL_SECURE_NO_WARNINGS",
+
+      "_HAS_EXCEPTIONS=0",
+      "_UNICODE",
+      "UNICODE",
+    ]
+    cflags += [
+      "/EHs-c-",
+      "/GR-",
+    ]
+
+    # The MSVC default value (1 MB) is not enough for parsing recursive C++
+    # templates in Clang.
+    ldflags = [ "/STACK:10000000" ]
+  }
+
+  # Warning setup.
+  if (host_os == "win" && !is_clang) {
+    cflags += [
+      # Suppress ''modifier' : used more than once' (__forceinline and inline).
+      "-wd4141",
+
+      # Suppress 'conversion from 'type1' to 'type2', possible loss of data'.
+      "-wd4244",
+
+      # Suppress 'conversion from 'size_t' to 'type', possible loss of data'.
+      "-wd4267",
+
+      # Suppress 'no matching operator delete found'.
+      "-wd4291",
+
+      # Suppress 'noexcept used with no exception handling mode specified'.
+      "-wd4577",
+
+      # Suppress 'destructor was implicitly defined as deleted'.
+      "-wd4624",
+
+      # Suppress 'unsafe mix of type <type> and type <type> in operation'.
+      "-wd4805",
+    ]
+  } else {
+    if (host_os == "win") {
+      cflags += [ "/W4" ]
+    } else {
+      cflags += [
+        "-Wall",
+        "-Wextra",
+      ]
+    }
+    cflags += [
+      "-Wno-unused-parameter",
+      "-Wstring-conversion",
+    ]
+    if (is_clang) {
+      cflags += [ "-Wdelete-non-virtual-dtor" ]
+    }
+    if (is_clang && use_goma) {
+      # goma converts all paths to lowercase on the server, breaking this
+      # warning.
+      cflags += [ "-Wno-nonportable-include-path" ]
+    }
+  }
+}
+
+config("llvm_code") {
+  include_dirs = [
+    "//llvm/include",
+    "$root_gen_dir/llvm/include",
+  ]
+}
+
+config("lld_code") {
+  include_dirs = [
+    "//lld/include",
+    "$root_gen_dir/lld/include",
+  ]
+}
+
+config("clang_code") {
+  include_dirs = [
+    "//clang/include",
+    "$root_gen_dir/clang/include",
+  ]
+}
+
+config("warn_covered_switch_default") {
+  if (is_clang) {
+    cflags = [ "-Wcovered-switch-default" ]
+  }
+}
+
+config("pthread_config") {
+  visibility = [ ":pthread" ]
+  libs = [ "pthread" ]
+}
+
+group("pthread") {
+  if (llvm_enable_threads && host_os != "win") {
+    public_configs = [ ":pthread_config" ]
+  }
+}

llvm/utils/gn/build/BUILDCONFIG.gn

@@ -0,0 +1,32 @@
+# All targets will get this list of configs by default.
+# Targets can opt out of a config by removing it from their local configs list.
+# If you're adding global flags and don't need targets to be able to opt out,
+# add the flags to compiler_defaults, not to a new config.
+_shared_binary_target_configs = [
+  "//llvm/utils/gn/build:compiler_defaults",
+  "//llvm/utils/gn/build:llvm_code",
+  "//llvm/utils/gn/build:warn_covered_switch_default",
+]
+
+# Apply that default list to the binary target types.
+set_defaults("executable") {
+  configs = _shared_binary_target_configs
+}
+set_defaults("loadable_module") {
+  configs = _shared_binary_target_configs
+}
+set_defaults("static_library") {
+  configs = _shared_binary_target_configs
+}
+set_defaults("shared_library") {
+  configs = _shared_binary_target_configs
+}
+set_defaults("source_set") {
+  configs = _shared_binary_target_configs
+}
+
+if (host_os == "win") {
+  set_default_toolchain("//llvm/utils/gn/build/toolchain:win")
+} else {
+  set_default_toolchain("//llvm/utils/gn/build/toolchain:unix")
+}

llvm/utils/gn/build/buildflags.gni

@@ -0,0 +1,10 @@
+declare_args() {
+  # Whether to build with debug information and without optimizations.
+  is_debug = false
+}
+
+# args that depend on other args must live in a later declare_args() block.
+declare_args() {
+  # Whether to enable assertions.
+  llvm_enable_assertions = true
+}

llvm/utils/gn/build/enable_threads.gni

@@ -0,0 +1,4 @@
+declare_args() {
+  # Whether to enable threading.
+  llvm_enable_threads = true
+}

llvm/utils/gn/build/mac_sdk.gni

@@ -0,0 +1,4 @@
+# Location of the mac sdk.
+# If that's not fixed, might want to shell out to xcrun at gn time to
+# retrieve this, but for now this seems to do the trick.
+mac_sdk_path = "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"

llvm/utils/gn/build/toolchain/BUILD.gn

@@ -0,0 +1,231 @@
+import("//llvm/utils/gn/build/toolchain/compiler.gni")
+
+declare_args() {
+  # If is_goma is true, the location of the goma client install.
+  if (host_os == "win") {
+    goma_dir = "c:\src\goma\goma-win64"
+  } else {
+    goma_dir = getenv("HOME") + "/goma"
+  }
+}
+
+toolchain("unix") {
+  cc = "cc"
+  cxx = "c++"
+
+  if (clang_base_path != "") {
+    cc = "$clang_base_path/bin/clang"
+    cxx = "$clang_base_path/bin/clang++"
+  }
+
+  ld = cxx  # Don't use goma wrapper for linking.
+  if (use_goma) {
+    cc = "$goma_dir/gomacc $cc"
+    cxx = "$goma_dir/gomacc $cxx"
+  }
+
+  tool("cc") {
+    depfile = "{{output}}.d"
+    command = "$cc -MMD -MF $depfile -o {{output}} -c {{source}} {{defines}} {{include_dirs}} {{cflags}} {{cflags_c}}"
+    depsformat = "gcc"
+    description = "CC {{output}}"
+    outputs = [
+      "{{source_out_dir}}/{{target_output_name}}.{{source_name_part}}.o",
+    ]
+  }
+
+  tool("cxx") {
+    depfile = "{{output}}.d"
+    command = "$cxx -MMD -MF $depfile -o {{output}} -c {{source}} {{defines}} {{include_dirs}} {{cflags}} {{cflags_cc}}"
+    depsformat = "gcc"
+    description = "CXX {{output}}"
+    outputs = [
+      "{{source_out_dir}}/{{target_output_name}}.{{source_name_part}}.o",
+    ]
+  }
+
+  tool("alink") {
+    if (host_os == "mac") {
+      command = "libtool -static -no_warning_for_no_symbols {{arflags}} -o {{output}} {{inputs}}"
+    } else {
+      # Remove the output file first so that ar doesn't try to modify the
+      # existing file.
+      command = "rm -f {{output}} && ar rcsDT {{arflags}} -o {{output}} {{inputs}}"
+    }
+    description = "AR {{output}}"
+    outputs = [
+      "{{output_dir}}/{{target_output_name}}.a",
+    ]
+    output_prefix = "lib"
+    default_output_dir = "{{root_out_dir}}/lib"
+  }
+
+  tool("solink") {
+    outfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    if (host_os == "mac") {
+      command = "$ld -shared {{ldflags}} -o $outfile {{libs}} {{inputs}}"
+      default_output_extension = ".dylib"
+    } else {
+      command =
+          "$ld -shared {{ldflags}} -Wl,-z,defs -o $outfile {{libs}} {{inputs}}"
+      default_output_extension = ".so"
+    }
+    description = "SOLINK $outfile"
+    outputs = [
+      outfile,
+    ]
+    lib_switch = "-l"
+    output_prefix = "lib"
+    default_output_dir = "{{root_out_dir}}/lib"
+  }
+
+  tool("solink_module") {
+    outfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    if (host_os == "mac") {
+      command = "$ld -shared {{ldflags}} -Wl,-flat_namespace -Wl,-undefined,suppress -o $outfile {{libs}} {{inputs}}"
+      default_output_extension = ".dylib"
+    } else {
+      command = "$ld -shared {{ldflags}} -o $outfile {{libs}} {{inputs}}"
+      default_output_extension = ".so"
+    }
+    description = "SOLINK $outfile"
+    outputs = [
+      outfile,
+    ]
+    lib_switch = "-l"
+    default_output_dir = "{{root_out_dir}}/lib"
+  }
+
+  tool("link") {
+    outfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    if (host_os == "mac") {
+      command = "$ld {{ldflags}} -o $outfile {{libs}} {{inputs}}"
+    } else {
+      command = "$ld {{ldflags}} -o $outfile {{libs}} -Wl,--start-group {{inputs}} -Wl,--end-group"
+    }
+    description = "LINK $outfile"
+    outputs = [
+      outfile,
+    ]
+    lib_switch = "-l"
+
+    # Setting this allows targets to override the default executable output by
+    # setting output_dir.
+    default_output_dir = "{{root_out_dir}}/bin"
+  }
+
+  tool("copy") {
+    command = "ln -f {{source}} {{output}} 2>/dev/null || (rm -rf {{output}} && cp -af {{source}} {{output}})"
+    description = "COPY {{source}} {{output}}"
+  }
+
+  tool("stamp") {
+    command = "touch {{output}}"
+    description = "STAMP {{output}}"
+  }
+}
+
+toolchain("win") {
+  cl = "cl"
+  link = "link"
+
+  if (clang_base_path != "") {
+    cl = "$clang_base_path/bin/clang-cl"
+    link = "$clang_base_path/bin/lld-link"
+  }
+
+  if (use_goma) {
+    cl = "$goma_dir/gomacc $cl"
+  }
+
+  tool("cc") {
+    command = "$cl /nologo /showIncludes /Fo{{output}} /c {{source}} {{defines}} {{include_dirs}} {{cflags}} {{cflags_c}}"
+    depsformat = "msvc"
+    description = "CC {{output}}"
+    outputs = [
+      "{{source_out_dir}}/{{target_output_name}}.{{source_name_part}}.obj",
+    ]
+  }
+
+  tool("cxx") {
+    command = "$cl /nologo /showIncludes /Fo{{output}} /c {{source}} {{defines}} {{include_dirs}} {{cflags}} {{cflags_cc}}"
+    depsformat = "msvc"
+    description = "CXX {{output}}"
+    outputs = [
+      "{{source_out_dir}}/{{target_output_name}}.{{source_name_part}}.obj",
+    ]
+  }
+
+  tool("alink") {
+    command = "lib /nologo {{arflags}} /out:{{output}} {{inputs}}"
+    description = "LIB {{output}}"
+    outputs = [
+      "{{output_dir}}/{{target_output_name}}.lib",
+    ]
+    default_output_dir = "{{root_out_dir}}/lib"
+  }
+
+  tool("solink") {
+    dllfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    libfile = "$dllfile.lib"
+    command = "$link /nologo /dll {{ldflags}} /out:$dllfile /implib:$libfile {{libs}} /pdb:$dllfile.pdb {{inputs}}"
+    description = "LINK $dllfile"
+    link_output = libfile
+    depend_output = libfile
+    runtime_outputs = [ dllfile ]
+    outputs = [
+      dllfile,
+      libfile,
+    ]
+    default_output_extension = ".dll"
+    restat = true
+
+    # Put dlls next to the executables in bin/ on Windows, since Windows
+    # doesn't have a configurable rpath. This matches initialization of
+    # module_dir to bin/ in AddLLVM.cmake's set_output_directory().
+    default_output_dir = "{{root_out_dir}}/bin"
+  }
+
+  # Plugins for opt and clang and so on don't work in LLVM's Windows build
+  # since the code doesn't have export annotations, but there are a few
+  # standalone loadable modules used for unit-testing LLVM's dynamic library
+  # loading code.
+  tool("solink_module") {
+    dllfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    command = "$link /nologo /dll {{ldflags}} /out:$dllfile {{libs}} /pdb:$dllfile.pdb {{inputs}}"
+    description = "LINK_MODULE $dllfile"
+    outputs = [
+      dllfile,
+    ]
+    runtime_outputs = outputs
+    default_output_extension = ".dll"
+
+    # No default_output_dir, all clients set output_dir.
+  }
+
+  tool("link") {
+    outfile = "{{output_dir}}/{{target_output_name}}{{output_extension}}"
+    command = "$link /nologo {{ldflags}} /out:$outfile {{libs}} /pdb:$outfile.pdb {{inputs}}"
+    description = "LINK $outfile"
+    outputs = [
+      outfile,
+    ]
+    default_output_extension = ".exe"
+
+    # Setting this allows targets to override the default executable output by
+    # setting output_dir.
+    default_output_dir = "{{root_out_dir}}/bin"
+  }
+
+  tool("copy") {
+    # GN hands out slash-using paths, but cmd's copy needs backslashes.
+    # Use cmd's %foo:a=b% substitution feature to convert.
+    command = "cmd /c set source=\"{{source}}\" & set output=\"{{output}}\" & call copy /Y %source:/=\% %output:\=/% > nul"
+    description = "COPY {{source}} {{output}}"
+  }
+
+  tool("stamp") {
+    command = "cmd /c type nul > {{output}}"
+    description = "STAMP {{output}}"
+  }
+}

llvm/utils/gn/build/toolchain/compiler.gni

@@ -0,0 +1,18 @@
+declare_args() {
+  # Whether to use goma (https://chromium.googlesource.com/infra/goma/client/)
+  use_goma = false
+
+  # Set this to a clang build directory. If set, that clang is used as compiler.
+  # goma only works with compiler binaries it knows about, so useful both for
+  # using a goma-approved compiler and for compiling clang with a locally-built
+  # clang in a different build directory.
+  # On Windows, setting this also causes lld-link to be used as linker.
+  # Example value: getenv("HOME") + "/src/llvm-build/Release+Asserts"
+  clang_base_path = ""
+}
+
+declare_args() {
+  # Set if the host compiler is clang.  On by default on Mac or if
+  # clang_base_path is set.
+  is_clang = host_os == "mac" || clang_base_path != ""
+}

llvm/utils/gn/secondary/BUILD.gn

@@ -0,0 +1,11 @@
+group("default") {
+  deps = [
+    "//llvm/lib/Demangle",
+  ]
+}
+
+# A pool called "console" in the root BUILD.gn is magic and represents ninja's
+# built-in console pool. (Requires a GN with `gn --version` >= 552353.)
+pool("console") {
+  depth = 1
+}

llvm/utils/gn/secondary/llvm/lib/Demangle/BUILD.gn

@@ -0,0 +1,9 @@
+static_library("Demangle") {
+  output_name = "LLVMDemangle"
+
+  sources = [
+    "ItaniumDemangle.cpp",
+    "MicrosoftDemangle.cpp",
+    "MicrosoftDemangleNodes.cpp",
+  ]
+}