[Dynamic obfuscation] change the value name 'obf_password' to 'obf_random_seed'

This commit is contained in:
jin-xiulang 2023-02-11 18:08:50 +08:00
parent 9a2d02f4b9
commit dd8f27a866
12 changed files with 156 additions and 151 deletions

View File

@ -37,4 +37,4 @@ mindspore.export
- **type** (str) - 混淆类型,目前支持动态混淆,即 'dynamic' 。
- **obf_ratio** (Union[str, float]) - 全模型算子的混淆比例,可取浮点数(0, 1]或者字符串 "small" 、 "medium" 、 "large" 。
- **customized_func** (function) - 在自定义函数模式下需要设置的Python函数用来控制混淆结构中的选择分支走向。它的返回值需要是bool类型且是恒定的用户可以参考不透明谓词进行设置。如果设置了 `customized_func` ,那么在使用 `load` 接口导入模型的时候,需要把这个函数也传入。
- **obf_password** (int) - 秘密口令用于password模式是一个大于0、小于等于int_64_max(9223372036854775807)的整数。如果用户设置了 `obf_password` ,那么在部署混淆模型的时候,需要在调用 :class:`mindspore.nn.GraphCell` 接口中传入 `obf_password` 。需要注意的是,如果用户同时设置了 `customized_func``obf_password` 那么password模式将会被采用。
- **obf_random_seed** (int) - 混淆随机种子,决定混淆分支的分布和权重混淆系数,是一个取值范围为(0, 9223372036854775807]的整数。如果用户设置了 `obf_random_seed` ,那么在部署混淆模型的时候,需要在调用 :class:`mindspore.nn.GraphCell` 接口中传入 `obf_random_seed` 。需要注意的是,如果用户同时设置了 `customized_func``obf_random_seed` ,那么后一种模式将会被采用。

View File

@ -14,7 +14,7 @@ mindspore.obfuscate_model
- **model_inputs** (list[Tensor]) - 模型的推理输入Tensor的值可以是随机的和使用 :func:`mindspore.export` 接口类似。
- **obf_ratio** (Union[str, float]) - 全模型算子的混淆比例,可取浮点数(0, 1]或者字符串"small"、"medium"、"large"。
- **customized_func** (function) - 在自定义函数模式下需要设置的Python函数用来控制混淆结构中的选择分支走向。它的返回值需要是bool类型且是恒定的用户可以参考不透明谓词进行设置。如果设置了 `customized_func` ,那么在使用 :func:`mindspore.load` 接口导入模型的时候,需要把这个函数也传入。
- **obf_password** (int) - 秘密口令用于password模式是一个大于0、小于等于int_64_max(9223372036854775807)的整数。如果用户设置了 `obf_password` ,那么在部署混淆模型的时候,需要在 :class:`mindspore.nn.GraphCell` 接口中传入 `obf_password` 。需要注意的是,如果用户同时设置了 `customized_func``obf_password` 那么password模式将会被采用。
- **obf_random_seed** (int) - 混淆随机种子,用于确定混淆分支的分布和权重混淆系数,是一个取值范围为(0, 9223372036854775807]的整数。如果用户设置了 `obf_random_seed` ,那么在部署混淆模型的时候,需要在 :class:`mindspore.nn.GraphCell` 接口中传入 `obf_random_seed` 。需要注意的是,如果用户同时设置了 `customized_func``obf_random_seed` ,那么后一种模式将会被采用。
- **kwargs** (dict) - 配置选项字典。
@ -28,6 +28,6 @@ mindspore.obfuscate_model
- **ValueError** - `original_model_path` 中的模型是已经混淆过的。
- **ValueError** - `obf_config` 没有提供 `save_model_path`
- **ValueError** - `obf_config` 没有提供 `obf_ratio`
- **ValueError** - `customized_func``obf_password` 都不在 `obf_config` 里面。
- **ValueError** - `obf_password` 的取值没有在(0, 9223372036854775807]内。
- **ValueError** - `customized_func``obf_random_seed` 都不在 `obf_config` 里面。
- **ValueError** - `obf_random_seed` 的取值没有在(0, 9223372036854775807]内。
- **ValueError** - `file_path` 不存在或者不以 '.mindir' 结尾。

View File

@ -1,7 +1,7 @@
mindspore.nn.GraphCell
======================
.. py:class:: mindspore.nn.GraphCell(graph, params_init=None, obf_password=None)
.. py:class:: mindspore.nn.GraphCell(graph, params_init=None, obf_random_seed=None)
运行从MindIR加载的计算图。
@ -10,7 +10,7 @@ mindspore.nn.GraphCell
参数:
- **graph** (FuncGraph) - 从MindIR加载的编译图。
- **params_init** (dict) - 需要在图中初始化的参数。key为参数名称类型为字符串value为 Tensor 或 Parameter。如果参数名在图中已经存在则更新其值如果不存在则忽略。默认值None。
- **obf_password** (int) - 用于动态混淆保护的password。动态混淆是一种模型保护方法,可以参考 :func:`mindspore.obfuscate_model` 。如果导入的 `graph` 是一个经过混淆的模型,那么 `obf_password` 应该要提供。 `obf_password` 的取值范围是(0, 9223372036854775807]。默认值None。
- **obf_random_seed** (Union[int, None]) - 用于动态混淆保护的混淆随机种子。动态混淆是一种模型保护方法,可以参考 :func:`mindspore.obfuscate_model` 。如果导入的 `graph` 是一个经过混淆的模型,那么须提供 `obf_random_seed``obf_random_seed` 的取值范围是(0, 9223372036854775807]。默认值None。
异常:
- **TypeError** - 如果图不是FuncGraph类型。

View File

@ -155,8 +155,7 @@ PYBIND11_MODULE(_c_expression, m) {
"Get graph proto string by specifying ir type.")
.def("get_obfuscate_func_graph_proto", &GraphExecutorPy::GetObfuscateFuncGraphProto, py::arg("phase") = py::str(""),
py::arg("incremental") = py::bool_(false), py::arg("obf_ratio") = py::float_(1.0),
py::arg("obf_pasword") = py::int_(0), py::arg("append_password") = py::int_(0),
"Get graph proto of dynamic-obfuscated model.")
py::arg("branch_control_input") = py::int_(0), "Get graph proto of dynamic-obfuscated model.")
.def("get_params", &GraphExecutorPy::GetParams, py::arg("phase") = py::str(""), "Get Parameters from graph")
.def("compile", &GraphExecutorPy::Compile, py::arg("obj"), py::arg("args"), py::arg("phase") = py::str(""),
py::arg("use_vm") = py::bool_(false), "Compile obj by executor.")
@ -212,8 +211,8 @@ PYBIND11_MODULE(_c_expression, m) {
py::arg("key_len") = py::int_(0), py::arg("dec_mode") = py::str("AES-GCM"),
py::arg("decrypt") = py::none(), py::arg("obfuscated") = py::bool_(false), "Load model as Graph.");
(void)m.def("dynamic_obfuscate_mindir", &mindspore::pipeline::DynamicObfuscateMindIR, py::arg("file_name"),
py::arg("obf_ratio"), py::arg("obf_password") = py::int_(0), py::arg("append_password") = py::int_(0),
py::arg("dec_key") = nullptr, py::arg("key_len") = py::int_(0), py::arg("dec_mode") = py::str("AES-GCM"),
py::arg("obf_ratio"), py::arg("branch_control_input") = py::int_(0), py::arg("dec_key") = nullptr,
py::arg("key_len") = py::int_(0), py::arg("dec_mode") = py::str("AES-GCM"),
"Obfuscate a mindir model by dynamic obfuscation.");
(void)m.def("init_cluster", &mindspore::distributed::Initialize, "Init Cluster");
(void)m.def("set_cluster_exit_with_exception", &mindspore::distributed::set_cluster_exit_with_exception,

View File

@ -587,15 +587,14 @@ py::bytes GraphExecutorPy::GetFuncGraphProto(const std::string &phase, const std
}
py::bytes GraphExecutorPy::GetObfuscateFuncGraphProto(const std::string &phase, const bool &incremental,
const float obf_ratio, const int obf_password,
const int append_password) {
const float obf_ratio, const int branch_control_input) {
FuncGraphPtr fg_ptr = GetFuncGraph(phase);
// obfuscate model
if (obf_password == 0) {
if (branch_control_input == 0) {
(void)mindspore::kernel::CustomizedOpaquePredicate::GetInstance().set_func_names();
MS_LOG(DEBUG) << "[GetObfuscateFuncGraphProto] set customized function names finished";
}
mindspore::DynamicObfuscator dynamic_obfuscator(obf_ratio, obf_password, append_password);
mindspore::DynamicObfuscator dynamic_obfuscator(obf_ratio, branch_control_input);
mindspore::FuncGraphPtr obfuscated_graph = dynamic_obfuscator.ObfuscateMindIR(fg_ptr);
std::string proto_str = GetBinaryProtoString(obfuscated_graph, incremental);
@ -1759,14 +1758,13 @@ FuncGraphPtr LoadMindIR(const std::string &file_name, const char *dec_key, const
return func_graph;
}
FuncGraphPtr DynamicObfuscateMindIR(const std::string &file_name, float obf_ratio, int obf_password,
int append_password, char *dec_key, const size_t key_len,
const std::string &dec_mode) {
if (obf_password == 0) {
FuncGraphPtr DynamicObfuscateMindIR(const std::string &file_name, float obf_ratio, int branch_control_input,
char *dec_key, const size_t key_len, const std::string &dec_mode) {
if (branch_control_input == 0) {
(void)mindspore::kernel::CustomizedOpaquePredicate::GetInstance().set_func_names();
MS_LOG(DEBUG) << "[DynamicObfuscateMindIR] set function names finished.";
}
mindspore::DynamicObfuscator dynamic_obfuscator(obf_ratio, obf_password, append_password);
mindspore::DynamicObfuscator dynamic_obfuscator(obf_ratio, branch_control_input);
MindIRLoader mindir_loader(false, reinterpret_cast<unsigned char *>(dec_key), key_len, dec_mode, false);
FuncGraphPtr func_graph = mindir_loader.LoadMindIR(file_name);
if (func_graph == nullptr) {

View File

@ -90,7 +90,7 @@ class GraphExecutorPy : public std::enable_shared_from_this<GraphExecutorPy> {
void SetGradGraph(const FuncGraphPtr &grad_graph, const std::string &phase);
py::bytes GetFuncGraphProto(const std::string &phase, const std::string &ir_type, const bool &incremental);
py::bytes GetObfuscateFuncGraphProto(const std::string &phase, const bool &incremental, const float obf_ratio,
const int obf_password, const int append_password);
const int branch_control_input);
#ifndef ENABLE_SECURITY
py::bytes GetOptimizeGraphProto(const std::string &phase);
#endif
@ -204,9 +204,8 @@ py::bytes PyEncrypt(char *plain_data, size_t plain_len, char *key, size_t key_le
py::bytes PyDecrypt(const std::string &encrypt_data_path, char *key, size_t key_len, const std::string &dec_mode);
bool PyIsCipherFile(const std::string &file_path);
void FinalizeCluster();
FuncGraphPtr DynamicObfuscateMindIR(const std::string &file_name, float obf_ratio, int obf_password,
int append_password, char *dec_key, const size_t key_len,
const std::string &dec_mode);
FuncGraphPtr DynamicObfuscateMindIR(const std::string &file_name, float obf_ratio, int branch_control_input,
char *dec_key, const size_t key_len, const std::string &dec_mode);
} // namespace pipeline
} // namespace mindspore

View File

@ -261,7 +261,7 @@ FuncGraphPtr DynamicObfuscator::ObfuscateMindIR(const FuncGraphPtr &func_graph)
if (subgraph_obf_num_ == 0) {
MS_LOG(WARNING)
<< "The model has not been obfuscated, which means obf_password or customized_func is not need to set.";
<< "The model has not been obfuscated, which means obf_random_seed or customized_func is not need to set.";
}
return func_graph;
}
@ -306,35 +306,24 @@ ObfCase DynamicObfuscator::ObfuscateOpCase(std::string obf_type) {
}
}
CNodePtr DynamicObfuscator::PasswordModeControl(FuncGraphPtr func_graph) {
CNodePtr DynamicObfuscator::RandomSeedModeControl(FuncGraphPtr func_graph) {
ShapeVector y_shape{1, 1};
tensor::TensorPtr y_tensor = std::make_shared<Tensor>(mindspore::kNumberTypeInt32, y_shape);
if (!has_build_appended_input) {
MS_LOG(INFO) << "Build parameter y_password and y_append.";
auto y = AddObfuscatedParam(func_graph);
y->set_name("y_password");
y->set_abstract(y_tensor->ToAbstract());
MS_LOG(INFO) << "Build parameter y_append.";
auto y_append = AddObfuscatedParam(func_graph);
y_append->set_name("y_append");
y_append->set_abstract(y_tensor->ToAbstract());
has_build_appended_input = true;
}
auto y = func_graph->GetParameterByName("y_password");
auto y_append = func_graph->GetParameterByName("y_append");
if (used_control_node_ == 0) {
// make add function node
mindspore::PrimitivePtr add_prim = mindspore::prim::kPrimAdd;
add_prim->set_attr("is_load", MakeValue(true));
mindspore::ValueNodePtr add_v_node = std::make_shared<mindspore::ValueNode>(add_prim);
(void)func_graph->AddValueNode(add_v_node);
CNodePtr add_c_node = func_graph->NewCNode({add_v_node, y, y_append});
add_c_node->set_abstract(y_tensor->ToAbstract());
// make equal function node
ValueNodePtr equal_v_node = std::make_shared<mindspore::ValueNode>(mindspore::prim::kPrimEqual);
(void)func_graph->AddValueNode(equal_v_node);
ValueNodePtr equal_compa_node = make_int_node(func_graph, obf_password_ + append_password_);
CNodePtr equal_c_node = func_graph->NewCNode({equal_v_node, add_c_node, equal_compa_node});
ValueNodePtr equal_compa_node = make_int_node(func_graph, branch_control_input_);
CNodePtr equal_c_node = func_graph->NewCNode({equal_v_node, y_append, equal_compa_node});
tensor::TensorPtr equal_tensor = std::make_shared<Tensor>(mindspore::kNumberTypeBool, y_shape);
equal_c_node->set_abstract(equal_tensor->ToAbstract());
(void)func_graph->AddNode(equal_c_node);
@ -347,12 +336,12 @@ CNodePtr DynamicObfuscator::PasswordModeControl(FuncGraphPtr func_graph) {
ValueNodePtr greater_v_node = std::make_shared<mindspore::ValueNode>(mindspore::prim::kPrimGreater);
(void)func_graph->AddValueNode(greater_v_node);
ValueNodePtr greater_compa_node = make_int_node(func_graph, comparison_int);
CNodePtr greater_c_node = func_graph->NewCNode({greater_v_node, y, greater_compa_node});
CNodePtr greater_c_node = func_graph->NewCNode({greater_v_node, y_append, greater_compa_node});
tensor::TensorPtr greater_tensor = std::make_shared<Tensor>(mindspore::kNumberTypeBool, y_shape);
greater_c_node->set_abstract(greater_tensor->ToAbstract());
(void)func_graph->AddNode(greater_c_node);
used_control_node_ += 1;
switch_branch_ = obf_password_ > comparison_int;
switch_branch_ = branch_control_input_ > comparison_int;
return greater_c_node;
}
@ -447,9 +436,9 @@ CNodePtr DynamicObfuscator::CustomOpModeControl(FuncGraphPtr func_graph, const A
}
CNodePtr DynamicObfuscator::GetControlNode(const FuncGraphPtr &func_graph, const AnfNodePtr &prev_node) {
if (obf_password_ != 0) {
if (branch_control_input_ != 0) {
MS_LOG(INFO) << "Run password mode.";
return PasswordModeControl(func_graph);
return RandomSeedModeControl(func_graph);
}
MS_LOG(INFO) << "Run customized function mode.";
if (prev_node != nullptr && prev_node->abstract() != nullptr) {
@ -528,7 +517,7 @@ void DynamicObfuscator::CheckDuplicatedParent(const AnfNodePtr &node) {
} else {
node_names_.push("-");
UpdateDict(node, true);
if (obf_password_ == 0) {
if (branch_control_input_ == 0) {
bool customized_func_result = mindspore::kernel::CustomizedOpaquePredicate::GetInstance().run_function(
static_cast<float>(1), static_cast<float>(1));
customized_func_results_.push_back(customized_func_result);
@ -905,7 +894,7 @@ void DynamicObfuscator::AddSwitchNode(FuncGraphPtr fg) {
mindspore::ValueNodePtr switch_v_node = std::make_shared<mindspore::ValueNode>(mindspore::prim::kPrimSwitch);
(void)fg->AddValueNode(switch_v_node);
mindspore::CNodePtr switch_c_node;
if (obf_password_ == 0) {
if (branch_control_input_ == 0) {
if (static_cast<int>(customized_func_results_.size()) <= used_control_node_) {
MS_LOG(ERROR) << "customized_func_results_ size is smaller than used_control_node_.";
}
@ -937,6 +926,26 @@ void DynamicObfuscator::AddSwitchNode(FuncGraphPtr fg) {
used_control_node_ = 0;
}
int GetNodeMaxNum(const AnfNodeSet nodes) {
int node_max_num = 0;
for (auto node : nodes) {
if (node != nullptr && node->isa<CNode>()) {
int op_num = get_op_num(node);
if (op_num > node_max_num) {
node_max_num = op_num;
}
}
}
return node_max_num;
}
bool DynamicObfuscator::IsValidOpNum(const int &current_num, const int &compa_num) const {
if (branch_control_input_ != 0) {
return true;
}
return current_num <= compa_num;
}
void DynamicObfuscator::SubGraphFakeBranch(FuncGraphPtr func_graph) {
if (func_graph == nullptr) {
MS_LOG(ERROR) << "Build fake sub-graph failed: FuncGraph is nullptr.";
@ -949,7 +958,8 @@ void DynamicObfuscator::SubGraphFakeBranch(FuncGraphPtr func_graph) {
auto all_nodes = mgr->all_nodes();
int node_nums = all_nodes.size();
int obfuscate_target_num = std::ceil(node_nums * obf_ratio_ / keyExpandRate);
int op_num = node_nums; // Initialize op_num to the maximum node number
int op_num = GetNodeMaxNum(all_nodes);
MS_LOG(INFO) << "Init op_num is: " << op_num;
std::vector<mindspore::AnfNodePtr> sorted_nodes;
for (auto node : all_nodes) {
MS_LOG(INFO) << "The last node name is: " << node->fullname_with_scope();
@ -963,10 +973,11 @@ void DynamicObfuscator::SubGraphFakeBranch(FuncGraphPtr func_graph) {
}
if (node->isa<CNode>()) {
std::string cnode_name = get_node_name(node);
MS_LOG(INFO) << "CNode name is: " << cnode_name;
int cur_op_num = get_op_num(node);
float dropout_rate = 0.1;
int dropout_rand = rand() % static_cast<int>(1.0 / dropout_rate);
if (IsTarget(cnode_name) && cur_op_num < op_num && dropout_rand != 0 &&
if (IsTarget(cnode_name) && IsValidOpNum(cur_op_num, op_num) && dropout_rand != 0 &&
(node_dict_.find(node->fullname_with_scope()) == node_dict_.cend())) {
UpdateDict(node, false);
op_num = cur_op_num;
@ -994,7 +1005,7 @@ void DynamicObfuscator::SubGraphFakeBranch(FuncGraphPtr func_graph) {
}
}
node_names_.pop();
if (obf_password_ == 0) {
if (branch_control_input_ == 0) {
(void)mindspore::kernel::CustomizedOpaquePredicate::GetInstance().init_calling_count();
}
(void)AddSwitchNode(func_graph);

View File

@ -30,8 +30,8 @@ namespace mindspore {
enum struct ObfCase : unsigned int { NotObfNode, OneInputNoWeightNode, OneInputWithWeightNode };
class COMMON_EXPORT DynamicObfuscator {
public:
DynamicObfuscator(const float obf_ratio, const int obf_password, const int append_password)
: obf_ratio_(obf_ratio), obf_password_(obf_password), append_password_(append_password) {}
DynamicObfuscator(const float obf_ratio, const int branch_control_input)
: obf_ratio_(obf_ratio), branch_control_input_(branch_control_input) {}
~DynamicObfuscator() = default;
@ -42,7 +42,7 @@ class COMMON_EXPORT DynamicObfuscator {
std::string ObfuscateOpType(const AnfNodePtr &node);
ObfCase ObfuscateOpCase(std::string obf_type);
CNodePtr GetControlNode(const FuncGraphPtr &func_graph, const AnfNodePtr &prev_node);
CNodePtr PasswordModeControl(FuncGraphPtr func_graph);
CNodePtr RandomSeedModeControl(FuncGraphPtr func_graph);
CNodePtr CustomOpModeControl(FuncGraphPtr func_graph, const AnfNodePtr &prev_node);
bool IsTarget(std::string &cnode_name);
@ -60,9 +60,9 @@ class COMMON_EXPORT DynamicObfuscator {
const AnfNodePtr &weights);
CNodePtr AddPartialBranch(FuncGraphPtr fg, FuncGraphPtr fg_sub, const std::vector<mindspore::CNodePtr> &nodes);
PrimitivePtr get_random_prim(const std::string &obf_type, const mindspore::CNodePtr &node);
bool IsValidOpNum(const int &current_num, const int &compa_num) const;
const float obf_ratio_ = 0.01;
const int obf_password_;
const int append_password_;
const int branch_control_input_;
bool has_build_appended_input = false;
std::vector<bool> customized_func_results_;
std::map<std::string, AnfNodePtr> node_dict_;

View File

@ -326,9 +326,8 @@ class _MindsporeFunctionExecutor:
if self.jit_config_dict:
enable_jit_level_o3 = self.jit_config_dict.get('jit_level') == "O3"
if (enable_ge and not enable_jit_level_o3) or (not enable_ge and enable_jit_level_o3):
raise RuntimeError("GE and jit_level=O3 should be used together, but "
"got MS_ENABLE_GE={}, jit_level={}".format(
os.getenv("MS_ENABLE_GE"), self.jit_config_dict.get('jit_level')))
raise RuntimeError("GE and jit_level=O3 should be used together, but got MS_ENABLE_GE={}, jit_level={}".
format(os.getenv("MS_ENABLE_GE"), self.jit_config_dict.get('jit_level')))
return output
@ -1476,16 +1475,16 @@ class _CellGraphExecutor:
def del_net_res(self, obj, net_id):
self._graph_executor.del_net_res(obj, net_id)
def _get_obf_pair_password(self):
def _get_branch_control_input(self):
if ('obf_ratio' not in self.obfuscate_config.keys()) or (
'obf_password' not in self.obfuscate_config.keys()):
raise ValueError("'obf_ratio' and 'obf_password' must be in obfuscate_config.")
obf_password = self.obfuscate_config.get('obf_password')
if obf_password == 0:
append_password = 0
'obf_random_seed' not in self.obfuscate_config.keys()):
raise ValueError("'obf_ratio' and 'obf_random_seed' must be in obfuscate_config.")
obf_random_seed = self.obfuscate_config.get('obf_random_seed')
if obf_random_seed == 0:
branch_control_input = 0
else:
obf_password, append_password = _generate_pair_password(obf_password)
return obf_password, append_password
branch_control_input = _generate_branch_control_input(obf_random_seed)
return branch_control_input
def _get_func_graph_proto(self, obj, exec_id, ir_type="onnx_ir", use_prefix=False, incremental=False):
"""Get graph proto from pipeline."""
@ -1494,10 +1493,10 @@ class _CellGraphExecutor:
if self._graph_executor.has_compiled(exec_id) is False:
return None
if self.obfuscate_config is not None:
obf_password, append_password = self._get_obf_pair_password()
branch_control_input = self._get_branch_control_input()
return self._graph_executor.get_obfuscate_func_graph_proto(exec_id, incremental,
self.obfuscate_config['obf_ratio'], obf_password,
append_password)
self.obfuscate_config['obf_ratio'],
branch_control_input)
return self._graph_executor.get_func_graph_proto(exec_id, ir_type, incremental)
def get_optimize_graph_proto(self, obj):
@ -1539,29 +1538,28 @@ def ms_memory_recycle():
_ms_memory_recycle()
def _generate_pair_password(obf_password):
"""Generate pair password for dynamic obfuscation in password mode."""
def _generate_branch_control_input(obf_random_seed):
"""Generate append network input for dynamic obfuscation in random seed mode."""
seed_max = 2 ** 32 - 1
int_max = 2 ** 31 - 1
np.random.seed(obf_password % seed_max)
np.random.seed(obf_random_seed % seed_max)
# generate a string as hash function inputs
word_repo = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghigklmnopqrstuvwxyz" + "0123456789"
repo_len = len(word_repo)
sha_string = ''
string_len = 1024*1024
string_len = 1024 * 1024
for _ in range(string_len):
rand_index = np.random.randint(0, repo_len)
sha_string += word_repo[rand_index]
# get hash result
sha_result = hashlib.sha256(sha_string.encode('utf-8')).hexdigest() # len is 64
append_password = 1
branch_control_input = 1
hex_base = 16
for item in sha_result:
if int(item, hex_base) > 0:
append_password *= int(item, hex_base)
append_password %= int_max
obf_password %= int_max
return obf_password, append_password
branch_control_input *= int(item, hex_base)
branch_control_input %= int_max
return branch_control_input
_cell_graph_executor = _CellGraphExecutor()

View File

@ -34,7 +34,7 @@ from mindspore._c_expression import init_pipeline, update_func_graph_hyper_param
from mindspore._checkparam import Validator
from mindspore.common import dtype as mstype
from mindspore.common.api import _cell_graph_executor, _pynative_executor, _get_args_for_run, cells_compile_cache
from mindspore.common.api import _generate_pair_password
from mindspore.common.api import _generate_branch_control_input
from mindspore.common.parameter import Parameter, ParameterTuple
from mindspore.common.tensor import Tensor
from mindspore.ops.operations import Cast
@ -1650,9 +1650,8 @@ class Cell(Cell_):
enable_ge = os.getenv("MS_ENABLE_GE") == '1'
enable_jit_level_o3 = self._jit_config_dict.get('jit_level') == "O3"
if (not enable_ge and enable_jit_level_o3) or (enable_ge and not enable_jit_level_o3):
raise RuntimeError("GE and jit_level=O3 should be used together, but "
"got MS_ENABLE_GE={}, jie_level={}".format(
os.getenv("MS_ENABLE_GE"), self.jit_config_dict.get('jit_level')))
raise RuntimeError("GE and jit_level=O3 should be used together, but got MS_ENABLE_GE={}, jie_level={}".
format(os.getenv("MS_ENABLE_GE"), self.jit_config_dict.get('jit_level')))
def flatten_weights(self, fusion_size=0):
"""
@ -2236,10 +2235,10 @@ class GraphCell(Cell):
The key is the parameter name whose type is str, and the value is a Tensor or Parameter.
If the parameter exists in the graph according to the name, update it's value.
If the parameter does not exist, ignore it. Default: None.
obf_password (int): The password used for dynamic obfuscation. "dynamic obfuscation" is used for model
protection, which can refer to :func:`mindspore.obfuscate_model`. If the input 'graph' is a
func_graph loaded from a mindir file obfuscated in password mode, then obf_password should be provided.
obf_password should be larger than zero and less or equal than int_64 (9223372036854775807). default: None.
obf_random_seed (Union[int, None]): The random seed used for dynamic obfuscation. "dynamic obfuscation" is
used for model protection, which can refer to :func:`mindspore.obfuscate_model`. If the input `graph` is
a func_graph loaded from a mindir file obfuscated with `obf_random_seed` , then `obf_random_seed` should be
provided. `obf_random_seed` should be in (0, 9223372036854775807]. default: None.
Raises:
TypeError: If the `graph` is not a FuncGraph.
@ -2269,22 +2268,22 @@ class GraphCell(Cell):
[4. 6. 4.]]]]
"""
def __init__(self, graph, params_init=None, obf_password=None):
def __init__(self, graph, params_init=None, obf_random_seed=None):
super(GraphCell, self).__init__(auto_prefix=True)
if not isinstance(graph, FuncGraph):
raise TypeError(f"For 'GraphCell', the argument 'graph' must be a FuncGraph loaded from MindIR, "
f"but got type {type(graph)}.")
self.graph = graph
self.obf_password = obf_password
if obf_password is not None:
if not isinstance(obf_password, int):
raise TypeError("'obf_password' must be int, but got {}.".format(type(obf_password)))
self.obf_random_seed = obf_random_seed
if obf_random_seed is not None:
if not isinstance(obf_random_seed, int):
raise TypeError("'obf_random_seed' must be int, but got {}.".format(type(obf_random_seed)))
int_64_max = 9223372036854775807
if obf_password <= 0 or obf_password > int_64_max:
if obf_random_seed <= 0 or obf_random_seed > int_64_max:
raise ValueError(
"'obf_password' must be larger than 0, and less or equal than int64 ({}),"
"but got {}.".format(int_64_max, obf_password))
self._obf_password, self._append_password = _generate_pair_password(self.obf_password)
"'obf_random_seed' must be larger than 0, and less or equal than int64 ({}),"
"but got {}.".format(int_64_max, obf_random_seed))
self._branch_control_input = _generate_branch_control_input(self.obf_random_seed)
params_init = {} if params_init is None else params_init
if not isinstance(params_init, dict):
raise TypeError(f"For 'GraphCell', the argument 'params_init' must be a dict, but got {type(params_init)}.")
@ -2304,11 +2303,10 @@ class GraphCell(Cell):
def __call__(self, *inputs):
self.phase = "graph_load_from_mindir"
self._add_attr("graph_load_from_mindir", self.graph)
if not self.obf_password:
if not self.obf_random_seed:
return self.compile_and_run(*inputs)
append_input_1 = Tensor((numpy.ones((1, 1)) * self._obf_password).astype(numpy.int32))
append_input_2 = Tensor((numpy.ones((1, 1)) * self._append_password).astype(numpy.int32))
return self.compile_and_run(*inputs, append_input_1, append_input_2)
append_input = Tensor((numpy.ones((1, 1)) * self._branch_control_input).astype(numpy.int32))
return self.compile_and_run(*inputs, append_input)
def _check_param_list_tuple(value):

View File

@ -45,7 +45,7 @@ from mindspore.common import dtype as mstype
from mindspore.common.api import _cell_graph_executor as _executor
from mindspore.common.api import _MindsporeFunctionExecutor
from mindspore.common.api import _get_parameter_layout
from mindspore.common.api import _generate_pair_password
from mindspore.common.api import _generate_branch_control_input
from mindspore.common.initializer import initializer, One
from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
@ -62,7 +62,6 @@ from mindspore.train._utils import read_proto
from mindspore._c_expression import load_mindir, _encrypt, _decrypt, _is_cipher_file, dynamic_obfuscate_mindir
from ..ops.operations._opaque_predicate_registry import add_opaque_predicate, clean_funcs
tensor_to_ms_type = {"Int8": mstype.int8, "UInt8": mstype.uint8, "Int16": mstype.int16, "UInt16": mstype.uint16,
"Int32": mstype.int32, "UInt32": mstype.uint32, "Int64": mstype.int64, "UInt64": mstype.uint64,
"Float16": mstype.float16, "Float32": mstype.float32, "Float64": mstype.float64,
@ -585,14 +584,15 @@ def _check_param_type(param_config, key, target_type, requested):
if key in param_config:
if not isinstance(param_config[key], target_type):
raise TypeError("The type of {} must be {}, but got {}.".format(key, target_type, type(param_config[key])))
if key == 'obf_password':
if key == 'obf_random_seed':
if param_config[key] > INT_64_MAX or param_config[key] <= 0:
raise ValueError(
"'obf_password' must be in (0, INT_64_MAX({})], but got {}.".format(INT_64_MAX, param_config[key]))
"'obf_random_seed' must be in (0, INT_64_MAX({})], but got {}.".format(INT_64_MAX,
param_config[key]))
return param_config[key]
if requested:
raise ValueError("The parameter {} is requested, but not got.".format(key))
if key == "obf_password":
if key == "obf_random_seed":
return 0
return None
@ -614,10 +614,10 @@ def _check_customized_func(customized_func):
def _check_obfuscate_params(obf_config):
"""Check obfuscation parameters, including obf_password, obf_ratio, customized_func"""
if 'obf_password' not in obf_config.keys() and 'customized_func' not in obf_config.keys():
"""Check obfuscation parameters, including obf_random_seed, obf_ratio, customized_func"""
if 'obf_random_seed' not in obf_config.keys() and 'customized_func' not in obf_config.keys():
raise ValueError(
"At least one of 'obf_password' or 'customized_func' must be set in obf_config, but got None of them.")
"At least one of 'obf_random_seed' or 'customized_func' must be set in obf_config, but got None of them.")
obfuscate_type = _check_param_type(obf_config, "type", str, False)
if obfuscate_type not in (None, "dynamic"):
raise ValueError("Only 'dynamic' type is supported by now, but got {}.".format(obfuscate_type))
@ -633,8 +633,8 @@ def _check_obfuscate_params(obf_config):
customized_funcs = []
if 'customized_func' in obf_config.keys():
customized_funcs.append(_check_customized_func(obf_config['customized_func']))
obf_password = _check_param_type(obf_config, "obf_password", int, False)
return obf_ratio, customized_funcs, obf_password
obf_random_seed = _check_param_type(obf_config, "obf_random_seed", int, False)
return obf_ratio, customized_funcs, obf_random_seed
def obfuscate_model(obf_config, **kwargs):
@ -658,10 +658,11 @@ def obfuscate_model(obf_config, **kwargs):
function needs to ensure that its result is constant for any input. Users can refer to opaque
predicates. If customized_func is set, then it should be passed to `load()` interface when loading
obfuscated model.
- obf_password (int): A password used for password mode, which should be in (0, 9223372036854775807]. If
obf_password is set, then it should be passed to `nn.GraphCell()` interface when loading obfuscated
model. It should be noted that at least one of 'customized_func' or 'obf_password' should be set, and
'obf_password' mode would be applied if both of them are set.
- obf_random_seed (int): The random seed used for determine the distribution of confusion branches and the
weight confusion coefficient, which should be in (0, 9223372036854775807]. If `obf_random_seed` is set,
then it should be passed to :class:`nn.GraphCell()` interface when loading obfuscated model. It should be
noted that at least one of `customized_func` or `obf_random_seed` should be set, and the latter mode
would be applied if both of them are set.
kwargs (dict): Configuration options dictionary.
@ -670,24 +671,24 @@ def obfuscate_model(obf_config, **kwargs):
Option: 'AES-GCM' | 'AES-CBC' | 'SM4-CBC'. Default: 'AES-GCM'.
Raises:
TypeError: If obf_config is not a dict.
ValueError: If enc_key is passed and enc_mode is not in ["AES-GCM", "AES-CBC", "SM4-CBC"].
ValueError: If original_model_path is not provided in obf_config.
ValueError: If the model saved in original_model_path has been obfuscated.
ValueError: If save_model_path is not provided in obf_config.
ValueError: If obf_ratio is not provided in obf_config.
ValueError: If both customized_func and obf_password are not provided in obf_config.
ValueError: If both obf_password is not in (0, 9223372036854775807].
ValueError: If file_path is not exist or file_path is not end with '.mindir'.
TypeError: If `obf_config` is not a dict.
ValueError: If `enc_key` is passed and `enc_mode` is not in ["AES-GCM", "AES-CBC", "SM4-CBC"].
ValueError: If `original_model_path` is not provided in `obf_config`.
ValueError: If the model saved in `original_model_path` has been obfuscated.
ValueError: If `save_model_path` is not provided in `obf_config`.
ValueError: If `obf_ratio` is not provided in `obf_config`.
ValueError: If both `customized_func` and `obf_random_seed` are not provided in `obf_config`.
ValueError: If `obf_random_seed` is not in (0, 9223372036854775807].
ValueError: If `original_model_path` is not exist or `original_model_path` is not end with '.mindir'.
Examples:
>>> obf_config = {'original_model_path': "./net.mindir",
... 'save_model_path': "./obf_net",
... 'model_inputs': [input1, ],
... 'obf_ratio': 0.1, 'obf_password': 173262358423}
... 'obf_ratio': 0.1, 'obf_random_seed': 173262358423}
>>> obfuscate_model(obf_config)
>>> obf_func = load("obf_net.mindir")
>>> obf_net = nn.GraphCell(obf_func, obf_password=173262358423)
>>> obf_net = nn.GraphCell(obf_func, obf_random_seed=173262358423)
>>> print(obf_net(input1).asnumpy())
"""
if not isinstance(obf_config, dict):
@ -707,18 +708,18 @@ def obfuscate_model(obf_config, **kwargs):
if -1 in item.shape:
raise ValueError(
"Dynamic shape input is not supported now, but got the shape of inputs: {}.".format(item.shape))
obf_ratio, customized_funcs, obf_password = _check_obfuscate_params(obf_config)
if customized_funcs and obf_password > 0:
logger.warning("Although 'customized_func' and 'obf_password' are set, the 'obf_password' mode would be"
" applied, remember to set 'obf_password' when loading obfuscated model.")
obf_ratio, customized_funcs, obf_random_seed = _check_obfuscate_params(obf_config)
if customized_funcs and obf_random_seed > 0:
logger.warning("Although 'customized_func' and 'obf_random_seed' are set, the 'obf_random_seed' mode would be"
" applied, remember to set 'obf_random_seed' when loading obfuscated model.")
if obf_password == 0: # apply customized_func mode
if obf_random_seed == 0: # apply customized_func mode
clean_funcs()
for func in customized_funcs:
add_opaque_predicate(func.__name__, func)
append_password = 0
branch_control_input = 0
else: # apply password mode
obf_password, append_password = _generate_pair_password(obf_password)
branch_control_input = _generate_branch_control_input(obf_random_seed)
if 'enc_key' in kwargs.keys():
enc_key = Validator.check_isinstance('enc_key', kwargs.get('enc_key'), bytes)
@ -729,18 +730,18 @@ def obfuscate_model(obf_config, **kwargs):
raise ValueError(
"Only MindIR files that encrypted with 'AES-GCM', 'AES-CBC' or 'SM4-CBC' is supported for"
"obfuscate_model(), but got {}.".format(enc_mode))
obf_graph = dynamic_obfuscate_mindir(file_name=file_path, obf_ratio=obf_ratio, obf_password=obf_password,
append_password=append_password, dec_key=enc_key, key_len=len(enc_key),
obf_graph = dynamic_obfuscate_mindir(file_name=file_path, obf_ratio=obf_ratio,
branch_control_input=branch_control_input, dec_key=enc_key,
key_len=len(enc_key),
dec_mode=enc_mode)
else:
obf_graph = dynamic_obfuscate_mindir(file_name=file_path, obf_ratio=obf_ratio, obf_password=obf_password,
append_password=append_password)
obf_graph = dynamic_obfuscate_mindir(file_name=file_path, obf_ratio=obf_ratio,
branch_control_input=branch_control_input)
obf_net = nn.GraphCell(obf_graph)
if obf_password != 0:
y_tensor = Tensor(np.ones((1, 1)).astype(np.int32))
if obf_random_seed != 0:
append_y_tensor = Tensor(np.ones((1, 1)).astype(np.int32))
model_inputs += [y_tensor, append_y_tensor]
model_inputs += [append_y_tensor,]
export(obf_net, *model_inputs, file_name=saved_path, file_format="MINDIR", **kwargs)
@ -1195,10 +1196,11 @@ def export(net, *inputs, file_name, file_format, **kwargs):
function needs to ensure that its result is constant for any input. Users can refer to opaque
predicates. If customized_func is set, then it should be passed to `load()` interface when loading
obfuscated model.
- obf_password (int): A password used for password mode, which should be in (0, 9223372036854775807]. If
obf_password is set, then it should be passed to `nn.GraphCell()` interface when loading obfuscated
model. It should be noted that at least one of 'customized_func' or 'obf_password' should be set, and
'obf_password' mode would be applied if both of them are set.
- obf_random_seed (int): The random seed used for determine the distribution of confusion branches and the
weight confusion coefficient, which should be in (0, 9223372036854775807]. If `obf_random_seed` is set,
then it should be passed to :class:`nn.GraphCell()` interface when loading obfuscated model. It should
be noted that at least one of `customized_func` or `obf_random_seed` should be set, and the latter mode
would be applied if both of them are set.
- incremental (bool): export MindIR incrementally.
@ -1483,16 +1485,16 @@ def _set_obfuscate_config(**kwargs):
raise ValueError(
"Only MindIR files that encrypted with 'AES-GCM', 'AES-CBC' or 'SM4-CBC' is supported for"
"obfuscation, but got {}.".format(enc_mode))
obf_ratio, customized_funcs, obf_password = _check_obfuscate_params(kwargs.get('obf_config'))
if customized_funcs and obf_password > 0:
logger.warning("Although 'customized_func' and 'obf_password' are set, the 'obf_password' mode would be"
" applied, remember to set 'obf_password' when loading obfuscated model.")
obf_ratio, customized_funcs, obf_random_seed = _check_obfuscate_params(kwargs.get('obf_config'))
if customized_funcs and obf_random_seed > 0:
logger.warning("Although 'customized_func' and 'obf_random_seed' are set, the 'obf_random_seed' mode would be"
" applied, remember to set 'obf_random_seed' when loading obfuscated model.")
if obf_password == 0: # apply customized_func mode
if obf_random_seed == 0: # apply customized_func mode
clean_funcs()
for func in customized_funcs:
add_opaque_predicate(func.__name__, func)
_executor.obfuscate_config = {'obf_ratio': obf_ratio, 'obf_password': obf_password}
_executor.obfuscate_config = {'obf_ratio': obf_ratio, 'obf_random_seed': obf_random_seed}
def _save_mindir(net, file_name, *inputs, **kwargs):

View File

@ -85,12 +85,12 @@ def test_obfuscate_model_password_mode():
# obfuscate model
obf_config = {"original_model_path": "net_1.mindir", "save_model_path": "./obf_net_1",
"model_inputs": [input_tensor], "obf_ratio": 0.8, "obf_password": 3423}
"model_inputs": [input_tensor], "obf_ratio": 0.8, "obf_random_seed": 3423}
obfuscate_model(obf_config)
# load obfuscated model, predict with right password
obf_graph_1 = load("obf_net_1.mindir")
obf_net_1 = nn.GraphCell(obf_graph_1, obf_password=3423)
obf_net_1 = nn.GraphCell(obf_graph_1, obf_random_seed=3423)
right_password_result = obf_net_1(input_tensor).asnumpy()
assert np.all(original_result == right_password_result)
@ -146,12 +146,12 @@ def test_export_password_mode():
original_result = net_3(input_tensor).asnumpy()
# obfuscate model
obf_config = {"obf_ratio": 0.8, "obf_password": 3423}
obf_config = {"obf_ratio": 0.8, "obf_random_seed": 3423}
export(net_3, input_tensor, file_name="obf_net_3", file_format="MINDIR", obf_config=obf_config)
# load obfuscated model, predict with right password
obf_graph_3 = load("obf_net_3.mindir")
obf_net_3 = nn.GraphCell(obf_graph_3, obf_password=3423)
obf_net_3 = nn.GraphCell(obf_graph_3, obf_random_seed=3423)
right_password_result = obf_net_3(input_tensor).asnumpy()
assert np.all(original_result == right_password_result)
@ -200,7 +200,7 @@ def test_wrong_file_format_input():
input_tensor = Tensor(np.ones((1, 1, 32, 32)).astype(np.float32))
# obfuscate model
obf_config = {"obf_ratio": 0.8, "obf_password": 3423}
obf_config = {"obf_ratio": 0.8, "obf_random_seed": 3423}
with pytest.raises(ValueError) as error_info:
export(net_5, input_tensor, file_name="obf_net_3", file_format="ONNX", obf_config=obf_config)
assert str(error_info.value) == "Dynamic obfuscation only support for MindIR format, but got ONNX format."