Previously to get around the issue that EKP is not present when restart test switching encryption from on to off and read encrypted data, EKP was made to start in simulation regardless of encryption knob. This PR revert that change, and instead force restart test not to change encryption knob, by passing previous encryption knob through restartInfo.ini file. Also since we don't allow downgrading an encrypted cluster to previous version, disable encryption in downgrade tests.
Also adding an assert to allow reading encrypted mutations only if encryption knob is on. We may reconsider allowing switching encryption on/off for existing cluster, but for now we don't allow it.
* Package inferred directories to the resulting tarball in ctest_to_joshua (except for the root source and build directories). This allows the API tests (which specify the workloads as a directory containing the workload specification files) to now work.
* Add bin/mkcert which is used by some tests.
* Rewrite fdb_c_shim_tests.py to specify its dependencies on command line.
In operator= overload we need to check if the current span which is being
overwritten is eligble for serialization. If so we must send to the tracer
via the g_tracer->trace(*this) call. Previously we were incorrectly checking
the function argument o's context for sampling. This resulted in loss traces
for the NAPI:readVersionBatcher spans.
Previously worker servers only register with cluster controller after local file recovery. This PR change to let worker servers register itself before local file recovery, but indicate that although it can server stateless roles, it cannot become storage or TLog server. After local file recovery, the worker will register again to indicate it can now become storage and Tlog.
The change fixes the deadlock issue with encryption, where when there's only stateful workers that can become master and EKP, those worker servers won't be able to register with cluster controller since no EKP is available to provide cipher keys to decrypt encrypted storage.
For now we have the change only when encryption is on, and will make it the default behavior once we think it is stable.
Xiaoxi Wang