foundationdb/design/special-key-space.md

Special-Key-Space

This document discusses why we need the proposed special-key-space framework. And for what problems the framework aims to solve and in what scenarios a developer should use it.

Motivation

Currently, there are several client functions implemented as FDB calls by passing through special keys(prefixed with \xff\xff). Below are all existing features:

• status/json: get("\xff\xff/status/json")
• cluster_file_path: get("\xff\xff/cluster_file_path)
• connection_string: get("\xff\xff/connection_string)
• worker_interfaces: getRange("\xff\xff/worker_interfaces", <any_key>)
• conflicting_keys: getRange("\xff\xff/transaction/conflicting_keys/", "\xff\xff/transaction/conflicting_keys/\xff")

At present, implementations are hard-coded and the pain points are obvious:

• Maintainability: As more features added, the hard-coded snippets are hard to maintain
• Granularity: It is impossible to scale up and down. For example, you want a cheap call like get("\xff\xff/status/json/<certain_field>") instead of calling status/json and parsing the results. On the contrary, sometime you want to aggregate results from several similar features like getRange("\xff\xff/transaction/, \xff\xff/transaction/\xff") to get all transaction related info. Both of them are not achievable at present.
• Consistency: While using FDB calls like get or getRange, the behavior that the result of get("\xff\xff/B") is not included in getRange("\xff\xff/A", "\xff\xff/C") is inconsistent with general FDB calls.

Consequently, the special-key-space framework wants to integrate all client functions using special keys(prefixed with \xff) and solve the pain points listed above.

When

If your feature is exposing information to clients and the results are easily formatted as key-value pairs, then you can use special-key-space to implement your client function.

How

If you choose to use, you need to implement a function class that inherits from SpecialKeyRangeReadImpl, which has an abstract method Future<RangeResult> getRange(ReadYourWritesTransaction* ryw, KeyRangeRef kr). This method can be treated as a callback, whose implementation details are determined by the developer. Once you fill out the method, register the function class to the corresponding key range. Below is a detailed example.

// Implement the function class,
// the corresponding key range is [\xff\xff/example/, \xff\xff/example/\xff)
class SKRExampleImpl : public SpecialKeyRangeReadImpl {
public:
    explicit SKRExampleImpl(KeyRangeRef kr): SpecialKeyRangeReadImpl(kr) {
        // Our implementation is quite simple here, the key-value pairs are formatted as:
        // \xff\xff/example/<country_name> : <capital_city_name>
        CountryToCapitalCity["USA"_sr] = "Washington, D.C."_sr;
        CountryToCapitalCity["UK"_sr] = "London"_sr;
        CountryToCapitalCity["Japan"_sr] = "Tokyo"_sr;
        CountryToCapitalCity["China"_sr] = "Beijing"_sr;
    }
    // Implement the getRange interface
    Future<RangeResult> getRange(ReadYourWritesTransaction* ryw,
                                            KeyRangeRef kr) const override {
        
        RangeResult result;
        for (auto const& country : CountryToCapitalCity) {
            // the registered range here: [\xff\xff/example/, \xff\xff/example/\xff]
            Key keyWithPrefix = country.first.withPrefix(range.begin);
            // check if any valid keys are given in the range
            if (kr.contains(keyWithPrefix)) {
                result.push_back(result.arena(), KeyValueRef(keyWithPrefix, country.second));
                result.arena().dependsOn(keyWithPrefix.arena());
            }
        }
        return result;
    }
private:
    std::map<Key, Value> CountryToCapitalCity;
};
// Instantiate the function object
// In development, you should have a function object pointer in DatabaseContext(DatabaseContext.h) and initialize in DatabaseContext's constructor(NativeAPI.actor.cpp)
const KeyRangeRef exampleRange("\xff\xff/example/"_sr, "\xff\xff/example/\xff"_sr);
SKRExampleImpl exampleImpl(exampleRange);
// Assuming the database handler is `cx`, register to special-key-space
// In development, you should register all function objects in the constructor of DatabaseContext(NativeAPI.actor.cpp)
cx->specialKeySpace->registerKeyRange(exampleRange, &exampleImpl);
// Now any ReadYourWritesTransaction associated with `cx` is able to query the info
state ReadYourWritesTransaction tr(cx);
// get
Optional<Value> res1 = wait(tr.get("\xff\xff/example/Japan"));
ASSERT(res1.present() && res.getValue() == "Tokyo"_sr);
// getRange
// Note: for getRange(key1, key2), both key1 and key2 should prefixed with \xff\xff
// something like getRange("normal_key", "\xff\xff/...") is not supported yet
RangeResult res2 = wait(tr.getRange("\xff\xff/example/U"_sr, "\xff\xff/example/U\xff"_sr));
// res2 should contain USA and UK
ASSERT(
    res2.size() == 2 &&
    res2[0].value == "London"_sr &&
    res2[1].value == "Washington, D.C."_sr
);

Module

We introduce this module concept after a discussion on cross module read on special-key-space. By default, range reads cover more than one module will not be allowed with special_keys_cross_module_read errors. In addition, range reads touch no modules will come with special_keys_no_module_found errors. The motivation here is to avoid unexpected blocking or errors happen in a wide-scope range read. In particular, you write code getRange("A", "Z") when all registered calls between [A, Z) happen locally, thus your code does not have any error-handling. However, if in the future, anyone register a new call in [A, Z) and sometimes throw errors like time_out(), then your original code is broken. The module is like a top-level directory where inside the module, calls are homogeneous. So we allow cross range read inside each module by default but cross module reads are forbidden. Right now, there are two modules available to use:

• TRANSACTION : \xff\xff/transaction/, \xff\xff/transaction0, all transaction related information like read_conflict_range, write_conflict_range, conflicting_keys.(All happen locally). Right now we have:
  • \xff\xff/transaction/conflicting_keys/, \xff\xff/transaction/conflicting_keys0 : conflicting keys that caused conflicts
  • \xff\xff/transaction/read_conflict_range/, \xff\xff/transaction/read_conflict_range0 : read conflict ranges of the transaction
  • \xff\xff/transaction/write_conflict_range/, \xff\xff/transaction/write_conflict_range0 : write conflict ranges of the transaction
• METRICS: \xff\xff/metrics/, \xff\xff/metrics0, all metrics like data-distribution metrics or healthy metrics are planned to put here. All need to call the rpc, so time_out error s may happen. Right now we have:
  • \xff\xff/metrics/data_distribution_stats/, \xff\xff/metrics/data_distribution_stats0 : stats info about data-distribution
• WORKERINTERFACE : \xff\xff/worker_interfaces/, \xff\xff/worker_interfaces0, which is compatible with previous implementation, thus should not be used to add new functions.

In addition, all singleKeyRanges are formatted as modules and cannot be used again. In particular, you should call get not getRange on these keys. Below are existing ones:

• STATUSJSON : \xff\xff/status/json
• CONNECTIONSTRING : \xff\xff/connection_string
• CLUSTERFILEPATH : \xff\xff/cluster_file_path