forked from OSchip/llvm-project
294 lines
12 KiB
Plaintext
294 lines
12 KiB
Plaintext
syncsource.py
|
|
|
|
OVERVIEW
|
|
|
|
The syncsource.py utility transfers groups of files between
|
|
computers. The primary use case is to enable developing LLVM project
|
|
software on one machine, transfer it efficiently to other machines ---
|
|
possibly of other architectures --- and test it there. syncsource.py
|
|
supports configurable, named source-to-destination mappings and has a
|
|
transfer agent plug-in architecture. The current distribution provides
|
|
an rsync-over-ssh transfer agent.
|
|
|
|
The primary benefits of using syncsource.py are:
|
|
|
|
* Provides a simple, reliable way to get a mirror copy of primary-
|
|
machine files onto several different destinations without concern
|
|
of compromising the patch during testing on different machines.
|
|
|
|
* Handles directory-mapping differences between two machines. For
|
|
LLDB, this is helpful when going between OS X and any other non-OS X
|
|
target system.
|
|
|
|
EXAMPLE WORKFLOW
|
|
|
|
This utility was developed in the context of working on the LLDB
|
|
project. Below we show the transfers we'd like to have happen,
|
|
and the configuration that supports it.
|
|
|
|
Workflow Example:
|
|
|
|
* Develop on OS X (primary machine)
|
|
* Test candidate changes on OS X.
|
|
* Test candidate changes on a Linux machine (machine-name: lldb-linux).
|
|
* Test candidate changes on a FreeBSD machine (machine-name: lldb-freebsd).
|
|
* Do check-ins from OS X machine.
|
|
|
|
Requirements:
|
|
|
|
* OS X machine requires the lldb source layout: lldb, lldb/llvm,
|
|
lldb/llvm/tools/clang. Note this is different than the canonical
|
|
llvm, llvm/tools/clang, llvm/tools/lldb layout that we'll want on
|
|
the Linux and FreeBSD test machines.
|
|
|
|
* Linux machine requires the llvm, llvm/tools/clang and
|
|
llvm/tools/lldb layout.
|
|
|
|
* FreeBSD machine requires the same layout as the llvm machine.
|
|
|
|
syncsource.py configuration in ~/.syncsourcerc:
|
|
|
|
# This is my configuration with a comment. Configuration
|
|
# files are JSON-based.
|
|
{ "configurations": [
|
|
# Here we have a collection of named configuration blocks.
|
|
# Configuration blocks can chain back to a parent by name.
|
|
{
|
|
# Every block has a name so it can be referenced from
|
|
# the command line or chained back to by a child block
|
|
# for sharing.
|
|
"name": "base_tot_settings",
|
|
|
|
# This directive lists the "directory ids" that we'll care
|
|
# about. If your local repository has additional directories
|
|
# for other projects that need to ride along, add them here.
|
|
# For defaulting purposes, it makes sense to name the
|
|
# directory IDs as the most likely name for the directory
|
|
# itself. For stock LLDB from top of tree, we generally only
|
|
# care about lldb, llvm and clang.
|
|
"dir_names": [ "llvm", "clang", "lldb" ],
|
|
|
|
# This section describes where the source files live on
|
|
# the primary machine. There should always be a base_dir
|
|
# entry, which indicates where in the local filesystem the
|
|
# projects are rooted. For each dir in dir_names, there
|
|
# should be either:
|
|
# 1. an entry named {dir-id}_dir (e.g. llvm_dir), which
|
|
# specifies the source directory for the given dir id
|
|
# relative to the base_dir entry, OR
|
|
# 2. no entry, in which case the directory is assumed to
|
|
# be the same as {dir-id}. In the example below, the
|
|
# base_dir-relative directory for the "lldb" dir-id is
|
|
# defaulted to being "lldb". That's exactly what
|
|
# we need in an OS X-style lldb dir layout.
|
|
"source": {
|
|
"base_dir": "~/work/lldb-tot",
|
|
"llvm_dir": "lldb/llvm",
|
|
"clang_dir": "lldb/llvm/tools/clang"
|
|
},
|
|
|
|
# source_excludes covers any exclusions that:
|
|
# * should be applied when copying files from the source
|
|
# * should be excluded from deletion on the destination
|
|
#
|
|
# By default, ".git", ".svn" and ".pyc" are added to
|
|
# all dir-id exclusions. The default excludes can be
|
|
# controlled by the syncsource.py --default-excludes
|
|
# option.
|
|
#
|
|
# Below, I have transfer of the lldb dir skip everything
|
|
# rooted at "/llvm" below the the lldb dir. This is
|
|
# because we want the source OS X lldb to move to
|
|
# a destination of {some-dest-root}/llvm/tools/lldb, and
|
|
# not have the OS-X-inverted llvm copy over with the lldb
|
|
# transfer portion. We'll see the complete picture of
|
|
# how this works when we get to specifying destinations
|
|
# later on in the config.
|
|
#
|
|
# We also exclude the "/build" and "/llvm-build" dir rooted in
|
|
# the OS X-side sources. The Xcode configuration on this
|
|
# OS X machine will dump lldb builds in the /build directory
|
|
# relative to the lldb dir, and it will build llvm+clang in
|
|
# the /llvm-build dir relative to the lldb dir.
|
|
#
|
|
# Note the first forward slash in "/build" indicates to the
|
|
# transfer agent that we only want to exclude the
|
|
# ~/work/lldb-tot/lldb/build dir, not just any file or
|
|
# directory named "build" somewhere underneath the lldb
|
|
# directory. Without the leading forward slash, any file
|
|
# or directory called build anywhere underneath the lldb dir
|
|
# will be excluded, which is definitely not what we want here.
|
|
#
|
|
# For the llvm dir, we do a source-side exclude for
|
|
# "/tools/clang". We manage the clang transfer as a separate
|
|
# entity, so we don't want the movement of llvm to also move
|
|
# clang.
|
|
#
|
|
# The llvm_dir exclusion of "/tools/lldb" is the first example
|
|
# of an exclude targeting a requirement of the destination
|
|
# side. Normally the transfer agent will delete anything on
|
|
# the destination that is not present on the source. It is
|
|
# trying to mirror, and ensure both sides have the same
|
|
# content. The source side of llvm on OS X does not have a
|
|
# "/tools/lldb", so at first this exclude looks non-sensical.
|
|
# But on the canonical destination layout, lldb lives in
|
|
# {some-dest-root}/llvm/tools/lldb. Without this exclude,
|
|
# the transfer agent would blow away the tools/lldb directory
|
|
# on the destination every time we transfer, and then have to
|
|
# copy the lldb dir all over again. For rsync+ssh, that
|
|
# totally would defeat the huge transfer efficiencies gained
|
|
# by using rsync in the first place.
|
|
#
|
|
# Note the overloading of both source and dest style excludes
|
|
# ultimately comes from the rsync-style exclude mechanism.
|
|
# If it wasn't for that, I would have separated source and
|
|
# dest excludes out better.
|
|
"source_excludes": {
|
|
"lldb_dir": ["/llvm", "/build", "/llvm-build"],
|
|
"llvm_dir": ["/tools/lldb", "/tools/clang"]
|
|
}
|
|
},
|
|
|
|
# Top of tree public, common settings for all destinations.
|
|
{
|
|
# The name for this config block.
|
|
"name": "common_tot",
|
|
|
|
# Here is our first chaining back to a parent config block.
|
|
# Any settings in "common_tot" not specified here are going
|
|
# to be retrieved from the parent.
|
|
"parent": "base_tot_settings",
|
|
|
|
# The transfer agent class to use. Right now, the only one
|
|
# available is this one here that uses rsync over ssh.
|
|
# If some other mechanism is needed to reach this destination,
|
|
# it can be specified here in full [[package.]module.]class form.
|
|
"transfer_class": "transfer.rsync.RsyncOverSsh",
|
|
|
|
# Specifies the destination-root-relative directories.
|
|
# Here our desination is rooted at:
|
|
# {some-yet-to-be-specified-destination-root} + "base_dir".
|
|
# In other words, each destination will have some kind of root
|
|
# for all relative file placement. We'll see those defined
|
|
# later, as they can change per destination machine.
|
|
# The block below describes the settings relative to that
|
|
# destination root.
|
|
#
|
|
# As before, each dir-id used in this configuration is
|
|
# expected to have either:
|
|
# 1. an entry named {dir-id}_dir (e.g. llvm_dir), which
|
|
# specifies the destination directory for the given dir id
|
|
# relative to the dest_root+base_dir entries, OR
|
|
# 2. no entry, in which case the directory is assumed to
|
|
# be the same as {dir-id}. In the example below, the
|
|
# dest_root+base_dir-relative directory for the "llvm" dir-id is
|
|
# defaulted to being "llvm". That's exactly what
|
|
# we need in a canonical llvm/clang/lldb setup on
|
|
# Linux/FreeBSD.
|
|
#
|
|
# Note we see the untangling of the OS X lldb-centric
|
|
# directory structure to the canonical llvm,
|
|
# llvm/tools/clang, llvm/tools/lldb structure below.
|
|
# We are mapping lldb into a subdirectory of the llvm
|
|
# directory.
|
|
#
|
|
# The transfer logic figures out which directories to copy
|
|
# first by finding the shortest destination absolute path
|
|
# and doing them in that order. For our case, this ensures
|
|
# llvm is copied over before lldb or clang.
|
|
"dest": {
|
|
"base_dir": "work/mirror/git",
|
|
"lldb_dir": "llvm/tools/lldb",
|
|
"clang_dir": "llvm/tools/clang"
|
|
}
|
|
},
|
|
|
|
# Describe the lldb-linux destination. With this,
|
|
# we're done with the mapping for transfer setup
|
|
# for the lldb-linux box. This configuration can
|
|
# be used either by:
|
|
# 1. having a parent "default" blockthat points to this one,
|
|
# which then gets used by default, or
|
|
# 2. using the --configuration/-c CONFIG option to
|
|
# specify using this name on the syncsource.py command line.
|
|
{
|
|
"name": "lldb-linux"
|
|
"parent": "common_tot",
|
|
|
|
# The ssh block is understood by the rsync+ssh transfer
|
|
# agent. Other agents would probably require different
|
|
# agent-specific details that they could read from
|
|
# other blocks.
|
|
"ssh": {
|
|
# This specifies the host name (or IP address) as would
|
|
# be used as the target for an ssh command.
|
|
"dest_host": "lldb-linux.example.com",
|
|
|
|
# root_dir specifies the global root directory for
|
|
# this destination. All destinations on this target
|
|
# will be in a directory that is built from
|
|
# root_dir + base_dir + {dir_id}_dir.
|
|
"root_dir" : "/home/tfiala",
|
|
|
|
# The ssh user is specified here.
|
|
"user": "tfiala",
|
|
|
|
# The ssh port is specified here.
|
|
"port": 22
|
|
}
|
|
},
|
|
|
|
# Describe the lldb-freebsd destination.
|
|
# Very similar to the lldb-linux one.
|
|
{
|
|
"name": "lldb-freebsd"
|
|
"parent": "common_tot",
|
|
"ssh": {
|
|
"dest_host": "lldb-freebsd.example.com",
|
|
# Specify a different destination-specific root dir here.
|
|
"root_dir" : "/mnt/ssd02/fialato",
|
|
"user": "fialato",
|
|
# The ssh port is specified here.
|
|
"port": 2022
|
|
}
|
|
},
|
|
|
|
# If a block named "default" exists, and if no configuration
|
|
# is specified on the command line, then the default block
|
|
# will be used. Use this block to point to the most common
|
|
# transfer destination you would use.
|
|
{
|
|
"name": "default",
|
|
"parent": "lldb-linux"
|
|
}
|
|
]
|
|
}
|
|
|
|
Using it
|
|
|
|
Now that we have a .syncsourcerc file set up, we can do a transfer.
|
|
The .syncsourcerc file will be searched for as follows, using the
|
|
first one that is found:
|
|
|
|
* First check the --rc-file RCFILE option. If this is specified
|
|
and doesn't exist, it will raise an error and quit.
|
|
|
|
* Check if the current directory has a .syncsourcerc file. If so,
|
|
use that.
|
|
|
|
* Use the .syncsourcerc file from the user's home directory.
|
|
|
|
Run the command:
|
|
python /path/to/syncsource.rc -c {configuration-name}
|
|
|
|
The -c {configuration-name} can be left off, in which case a
|
|
configuration with the name 'default' will be used.
|
|
|
|
After that, the transfer will occur. With the rsync-over-ssh
|
|
transfer agent, one rsync per dir-id will be used. rsync output
|
|
is redirected to the console.
|
|
|
|
FEEDBACK
|
|
|
|
Feel free to pass feedback along to Todd Fiala (todd.fiala@gmail.com).
|