Compare commits

...

27 Commits

Author SHA1 Message Date
Jermy Li 12ffce2cc3
Update README.md 2021-12-31 14:23:31 +08:00
zhangyi51 1870ecf3a3 upgrade log4j 2.17.0
Change-Id: I2c7ceb113bae25e02f6fb7e9df02c6193eccd920
2021-12-21 19:49:12 +08:00
zhangyi51 025e9ba5ab fix log4j error
Change-Id: I991e975d254bb3522070a52d73326c6ea99424ab
2021-12-13 16:20:47 +08:00
imbajin 94ea15ec1c refact: unify kout/kneighbor get method with post (#1470) 2021-05-28 16:18:44 +08:00
zhangyi51 11d2c927ca keep LimitIterator in GraphTransaction to avoid upgrade common version from 1.8.1 to 1.8.4
Change-Id: If29ecf55f02d9e8bbb351052d1760b69e575ded8
2021-05-27 20:19:13 +08:00
imbajin 04274279c3 refact: remove source from kneighbor/kout results & unify 'degree ' error info (#1463)
remove source vertex from kout algorithm
unify 'degree ' error info & remove source from kneighbor/kout result
2021-05-27 20:18:37 +08:00
imbajin 4de56ae74e refact: let kout degree applied to all labels & remove source vertex from kneighbor default results (#1459) 2021-05-27 20:18:16 +08:00
imbajin 5a3d925c64 fix: replace '+' to '%2B' to handle page strs encode (#1437)
Change-Id: I58aa9d84faf14104d0fb7eef25a21c6317524cd0
2021-05-27 20:18:13 +08:00
imbajin 906daa40cb refact: unify naming of degree for oltp algorithms (#1433)
* refact: remove redundant packaging
* refact: keep compatible with degree in 'step'
* update api version to 0.61

Change-Id: Ibbfcaab820eed9d4131a5544899a5e3829729201
2021-05-27 20:17:35 +08:00
imbajin 5150bb8167 refact: improve error info of edge-update-api (#1443) 2021-05-27 20:17:34 +08:00
zyxxoo 03dce6f42e use NumericUtil to convert weight value (#1438) 2021-05-27 20:17:34 +08:00
zhoney e765a55894 fix kout/kneighbor default limit from 10 to 10000000 (#1436)
Change-Id: I627d174311186de5b9cc86fb1e23c155e4affbe3
2021-05-27 20:17:34 +08:00
zyxxoo eb54c27dd7 Fix paths get and post method dont same (#1435) 2021-05-27 20:17:34 +08:00
zyxxoo 920254523f Fix forced to Double and optimization error hint for top range (#1432) 2021-05-27 20:17:34 +08:00
zyxxoo ddea5a8a8e Fix fusiform similarity api return all vertices when similars is empty (#1434) 2021-05-27 20:17:34 +08:00
Jermy Li 8fa020d963 fix ScyllaDB: lost page results due to not fetched the entire page (#1407)
fix #1340

* return page offset if not return local fetched data to users

Change-Id: I8abe05cbdc95c28bc384e73d554a30300042f3bb
2021-05-27 20:17:34 +08:00
Linary 6d9d21fc1d Fix some license params (#1420)
Change-Id: I89943227add35b5fdd6032e8dbb4a3ef3e3826d2
2021-05-27 20:17:34 +08:00
zhoney 2436890a04 fix allowed append vl pk or el sk as nullable props (#1406)
Change-Id: I9514e088718414809f9b2e1477f358648a581df4
2021-05-27 20:17:34 +08:00
Coderzc e42f56ab07 Fix macos free_memory calculate by using vm_stat (#1396) 2021-03-29 15:10:19 +08:00
Linary 61466d6a61 Fix inappropriate shard will cause base64 decode error (#1383)
* Fix shard start is 0 will cause base64 decode error

Change-Id: Ibc90df4e792b00222d41806fee769082545dea40
2021-03-29 15:10:19 +08:00
Jermy Li a495eeab69 avoid deleteRange() as possible for performance (#1375)
Change-Id: I1667e062b8a6a99c52d534a314a0af4bfecb59a0
2021-03-29 15:10:19 +08:00
Jermy Li 1da70365a9 fix task: atomic update/get fields and re-schedule (#1361)
* atomic update and get task status&result
* call resubmitTask() when re-schedule task

Change-Id: I2efa9fd4979492c6901e1ae11fbaf2be451f0354
2021-03-29 15:10:19 +08:00
Jermy Li 0f3be142aa fix task result with path/tree can't be serialized (#1351)
Change-Id: I5f740632f49b1305785a118a27236049cc71af41
2021-02-05 16:53:26 +08:00
Linary d9d06969da Add TarLz4Util to improve perf for comressing snapshot (#1336)
Change-Id: I5ad156ecba757af0131b1096a956e4c3ca5872cd
2021-02-05 16:53:26 +08:00
Jermy Li 7578e8ff66 auth-delete and role-get APIs need to check if the graph exists (#1338)
Change-Id: I5b76437bd68b1dbbe7bda04c796b9b815104fe20
2021-02-05 16:53:26 +08:00
Jermy Li 4fbff6a1bd vertex/edge get api use queryVertex/queryEdge instead of iterator (#1345)
also improve some code

Change-Id: I0048c45c9a00bdc81a1b55129a6589043733ef9d
2021-02-05 16:53:26 +08:00
zhoney adeb01c893 fix gremlin job result size gt cassandra limit and lt hugegraph limit (#1334)
Change-Id: Ic2a246facffca683052bb56bf76ad21b5413d94a
2021-02-05 16:53:26 +08:00
82 changed files with 1412 additions and 448 deletions

View File

@ -1,8 +1,8 @@
# HugeGraph
[![License](https://img.shields.io/badge/license-Apache%202-0E78BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![Build Status](https://travis-ci.org/hugegraph/hugegraph.svg?branch=master)](https://travis-ci.org/hugegraph/hugegraph)
[![codecov](https://codecov.io/gh/hugegraph/hugegraph/branch/master/graph/badge.svg)](https://codecov.io/gh/hugegraph/hugegraph)
[![Build Status](https://travis-ci.org/hugegraph/hugegraph.svg?branch=release-0.11)](https://travis-ci.org/hugegraph/hugegraph)
[![codecov](https://codecov.io/gh/hugegraph/hugegraph/branch/release-0.11/graph/badge.svg)](https://codecov.io/gh/hugegraph/hugegraph)
HugeGraph is a fast-speed and highly-scalable [graph database](https://en.wikipedia.org/wiki/Graph_database). Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability. As compliance to [Apache TinkerPop 3](https://tinkerpop.apache.org/) framework, various complicated graph queries can be accomplished through [Gremlin](https://tinkerpop.apache.org/gremlin.html)(a powerful graph traversal language).

View File

@ -149,6 +149,8 @@ public class AccessAPI extends API {
@PathParam("id") String id) {
LOG.debug("Graph [{}] delete access: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
try {
manager.userManager().deleteAccess(UserAPI.parseId(id));
} catch (NotFoundException e) {

View File

@ -148,6 +148,8 @@ public class BelongAPI extends API {
@PathParam("id") String id) {
LOG.debug("Graph [{}] delete belong: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
try {
manager.userManager().deleteBelong(UserAPI.parseId(id));
} catch (NotFoundException e) {

View File

@ -134,6 +134,8 @@ public class GroupAPI extends API {
@PathParam("id") String id) {
LOG.debug("Graph [{}] delete group: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
try {
manager.userManager().deleteGroup(IdGenerator.of(id));
} catch (NotFoundException e) {

View File

@ -134,6 +134,9 @@ public class TargetAPI extends API {
@PathParam("graph") String graph,
@PathParam("id") String id) {
LOG.debug("Graph [{}] delete target: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
try {
manager.userManager().deleteTarget(UserAPI.parseId(id));
} catch (NotFoundException e) {

View File

@ -136,6 +136,8 @@ public class UserAPI extends API {
@PathParam("id") String id) {
LOG.debug("Graph [{}] get user role: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
HugeUser user = manager.userManager().getUser(IdGenerator.of(id));
return manager.userManager().rolePermission(user).toJson();
}
@ -149,6 +151,8 @@ public class UserAPI extends API {
@PathParam("id") String id) {
LOG.debug("Graph [{}] delete user: {}", graph, id);
@SuppressWarnings("unused") // just check if the graph exists
HugeGraph g = graph(manager, graph);
try {
manager.userManager().deleteUser(IdGenerator.of(id));
} catch (NotFoundException e) {

View File

@ -148,7 +148,7 @@ public class BatchAPI extends API {
} else if (oldElement.property(key).isPresent() &&
newElement.properties.get(key) == null) {
// If new property is null & old is present, use old property
newElement.properties.put(key, oldElement.property(key).value());
newElement.properties.put(key, oldElement.value(key));
}
}
}

View File

@ -233,15 +233,15 @@ public class EdgeAPI extends BatchAPI {
if (jsonEdge.id != null) {
E.checkArgument(id.equals(jsonEdge.id),
"The ids are different between url('%s') and " +
"request body('%s')", id, jsonEdge.id);
"The ids are different between url and " +
"request body ('%s' != '%s')", id, jsonEdge.id);
}
// Parse action param
boolean append = checkAndParseAction(action);
HugeGraph g = graph(manager, graph);
HugeEdge edge = (HugeEdge) g.edges(id).next();
HugeEdge edge = (HugeEdge) g.edge(id);
EdgeLabel edgeLabel = edge.schemaLabel();
for (String key : jsonEdge.properties.keySet()) {
@ -458,8 +458,11 @@ public class EdgeAPI extends BatchAPI {
HugeVertex.getIdValue(newEdge.target));
if (newEdge.id != null) {
E.checkArgument(edgeId.equals(newEdge.id),
"The sort key values either be null " +
"or equal to origin when specified edge id");
"The ids are different between server and " +
"request body ('%s' != '%s'). And note the sort " +
"key values should either be null or equal to " +
"the origin value when specified edge id",
edgeId, newEdge.id);
}
return edgeId;
}

View File

@ -201,7 +201,7 @@ public class VertexAPI extends BatchAPI {
boolean append = checkAndParseAction(action);
HugeGraph g = graph(manager, graph);
HugeVertex vertex = (HugeVertex) g.vertices(id).next();
HugeVertex vertex = (HugeVertex) g.vertex(id);
VertexLabel vertexLabel = vertex.schemaLabel();
for (String key : jsonVertex.properties.keySet()) {

View File

@ -22,7 +22,7 @@ package com.baidu.hugegraph.api.traversers;
import java.util.List;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import javax.inject.Singleton;
import javax.ws.rs.DefaultValue;
@ -66,7 +66,7 @@ public class AllShortestPathsAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("skip_degree")
@DefaultValue("0") long skipDegree,
@QueryParam("capacity")
@ -75,7 +75,7 @@ public class AllShortestPathsAPI extends API {
"direction {}, edge label {}, max depth '{}', " +
"max degree '{}', skipped degree '{}' and capacity '{}'",
graph, source, target, direction, edgeLabel, depth,
degree, skipDegree, capacity);
maxDegree, skipDegree, capacity);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Id targetId = VertexAPI.checkAndParseVertexId(target);
@ -88,7 +88,7 @@ public class AllShortestPathsAPI extends API {
ImmutableList.of(edgeLabel);
HugeTraverser.PathSet paths = traverser.allShortestPaths(
sourceId, targetId, dir, edgeLabels,
depth, degree, skipDegree, capacity);
depth, maxDegree, skipDegree, capacity);
return manager.serializer(g).writePaths("paths", paths, false);
}
}

View File

@ -19,7 +19,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_SKIP_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.NO_LIMIT;
@ -48,6 +48,7 @@ import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.codahale.metrics.annotation.Timed;
import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.collect.ImmutableMap;
@ -74,7 +75,8 @@ public class CountAPI extends API {
"The steps of request can't be null or empty");
E.checkArgumentNotNull(request.dedupSize == NO_LIMIT ||
request.dedupSize >= 0L,
"The dedupSize of request must >= 0, but got '%s'",
"The dedup size of request " +
"must >= 0 or == -1, but got: '%s'",
request.dedupSize);
HugeGraph g = graph(manager, graph);
@ -109,7 +111,7 @@ public class CountAPI extends API {
@Override
public String toString() {
return String.format("CountRequest{source=%s,steps=%s," +
"contains_traversed=%s,dedupSize=%s}",
"containsTraversed=%s,dedupSize=%s}",
this.source, this.steps,
this.containsTraversed, this.dedupSize);
}
@ -123,22 +125,24 @@ public class CountAPI extends API {
public List<String> labels;
@JsonProperty("properties")
public Map<String, Object> properties;
@JsonProperty("degree")
public long degree = Long.valueOf(DEFAULT_DEGREE);
@JsonAlias("degree")
@JsonProperty("max_degree")
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("skip_degree")
public long skipDegree = Long.valueOf(DEFAULT_SKIP_DEGREE);
public long skipDegree = Long.parseLong(DEFAULT_SKIP_DEGREE);
@Override
public String toString() {
return String.format("Step{direction=%s,labels=%s,properties=%s" +
"degree=%s,skipDegree=%s}",
"maxDegree=%s,skipDegree=%s}",
this.direction, this.labels, this.properties,
this.degree, this.skipDegree);
this.maxDegree, this.skipDegree);
}
private EdgeStep jsonToStep(HugeGraph graph) {
return new EdgeStep(graph, this.direction, this.labels,
this.properties, this.degree, this.skipDegree);
this.properties, this.maxDegree,
this.skipDegree);
}
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import javax.inject.Singleton;
@ -64,7 +64,7 @@ public class CrosspointsAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity,
@QueryParam("limit")
@ -73,7 +73,7 @@ public class CrosspointsAPI extends API {
"with direction '{}', edge label '{}', max depth '{}', " +
"max degree '{}', capacity '{}' and limit '{}'",
graph, source, target, direction, edgeLabel,
depth, degree, capacity, limit);
depth, maxDegree, capacity, limit);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Id targetId = VertexAPI.checkAndParseVertexId(target);
@ -83,7 +83,7 @@ public class CrosspointsAPI extends API {
PathsTraverser traverser = new PathsTraverser(g);
HugeTraverser.PathSet paths = traverser.paths(sourceId, dir, targetId,
dir, edgeLabel, depth,
degree, capacity, limit);
maxDegree, capacity, limit);
return manager.serializer(g).writePaths("crosspoints", paths, true);
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import java.util.ArrayList;
@ -53,6 +53,7 @@ import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.codahale.metrics.annotation.Timed;
import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty;
@Path("graphs/{graph}/traversers/customizedcrosspoints")
@ -136,9 +137,9 @@ public class CustomizedCrosspointsAPI extends API {
@JsonProperty("path_patterns")
public List<PathPattern> pathPatterns;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_PATHS_LIMIT);
@JsonProperty("with_path")
public boolean withPath = false;
@JsonProperty("with_vertex")
@ -173,24 +174,25 @@ public class CustomizedCrosspointsAPI extends API {
public List<String> labels;
@JsonProperty("properties")
public Map<String, Object> properties;
@JsonProperty("degree")
public long degree = Long.valueOf(DEFAULT_DEGREE);
@JsonAlias("degree")
@JsonProperty("max_degree")
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("skip_degree")
public long skipDegree = 0L;
@Override
public String toString() {
return String.format("Step{direction=%s,labels=%s,properties=%s," +
"degree=%s,skipDegree=%s}",
"maxDegree=%s,skipDegree=%s}",
this.direction, this.labels, this.properties,
this.degree, this.skipDegree);
this.maxDegree, this.skipDegree);
}
private CustomizedCrosspointsTraverser.Step jsonToStep(HugeGraph g) {
return new CustomizedCrosspointsTraverser.Step(g, this.direction,
this.labels,
this.properties,
this.degree,
this.maxDegree,
this.skipDegree);
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_SAMPLE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_WEIGHT;
@ -56,6 +56,7 @@ import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.codahale.metrics.annotation.Timed;
import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty;
@Path("graphs/{graph}/traversers/customizedpaths")
@ -136,9 +137,9 @@ public class CustomizedPathsAPI extends API {
@JsonProperty("sort_by")
public SortBy sortBy;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_PATHS_LIMIT);
@JsonProperty("with_vertex")
public boolean withVertex = false;
@ -160,31 +161,32 @@ public class CustomizedPathsAPI extends API {
public List<String> labels;
@JsonProperty("properties")
public Map<String, Object> properties;
@JsonProperty("degree")
public long degree = Long.valueOf(DEFAULT_DEGREE);
@JsonAlias("degree")
@JsonProperty("max_degree")
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("skip_degree")
public long skipDegree = 0L;
@JsonProperty("weight_by")
public String weightBy;
@JsonProperty("default_weight")
public double defaultWeight = Double.valueOf(DEFAULT_WEIGHT);
public double defaultWeight = Double.parseDouble(DEFAULT_WEIGHT);
@JsonProperty("sample")
public long sample = Long.valueOf(DEFAULT_SAMPLE);
public long sample = Long.parseLong(DEFAULT_SAMPLE);
@Override
public String toString() {
return String.format("Step{direction=%s,labels=%s,properties=%s," +
"degree=%s,skipDegree=%s," +
"maxDegree=%s,skipDegree=%s," +
"weightBy=%s,defaultWeight=%s,sample=%s}",
this.direction, this.labels, this.properties,
this.degree, this.skipDegree,
this.maxDegree, this.skipDegree,
this.weightBy, this.defaultWeight,
this.sample);
}
private WeightedEdgeStep jsonToStep(HugeGraph g) {
return new WeightedEdgeStep(g, this.direction, this.labels,
this.properties, this.degree,
this.properties, this.maxDegree,
this.skipDegree, this.weightBy,
this.defaultWeight, this.sample);
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.NO_LIMIT;
@ -76,9 +76,9 @@ public class FusiformSimilarityAPI extends API {
E.checkArgument(request.minNeighbors > 0,
"The min neighbor count must be > 0, but got: %s",
request.minNeighbors);
E.checkArgument(request.degree > 0 || request.degree == NO_LIMIT,
"The degree of request must be > 0, but got: %s",
request.degree);
E.checkArgument(request.maxDegree > 0L || request.maxDegree == NO_LIMIT,
"The max degree of request must be > 0 or == -1, " +
"but got: %s", request.maxDegree);
E.checkArgument(request.alpha > 0 && request.alpha <= 1.0,
"The alpha of request must be in range (0, 1], " +
"but got '%s'", request.alpha);
@ -108,13 +108,13 @@ public class FusiformSimilarityAPI extends API {
request.minNeighbors, request.alpha,
request.minSimilars, request.top,
request.groupProperty, request.minGroups,
request.degree, request.capacity, request.limit,
request.withIntermediary);
request.maxDegree, request.capacity,
request.limit, request.withIntermediary);
CloseableIterator.closeIterator(sources);
Iterator<Vertex> iterator = QueryResults.emptyIterator();
if (request.withVertex) {
if (request.withVertex && !result.isEmpty()) {
iterator = g.vertices(result.vertices().toArray());
}
return manager.serializer(g).writeSimilars(result, iterator);
@ -141,11 +141,11 @@ public class FusiformSimilarityAPI extends API {
@JsonProperty("min_groups")
public int minGroups;
@JsonProperty("max_degree")
public long degree = Long.valueOf(DEFAULT_DEGREE);
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_PATHS_LIMIT);
@JsonProperty("with_intermediary")
public boolean withIntermediary = false;
@JsonProperty("with_vertex")
@ -157,13 +157,13 @@ public class FusiformSimilarityAPI extends API {
"label=%s,direction=%s,minNeighbors=%s," +
"alpha=%s,minSimilars=%s,top=%s," +
"groupProperty=%s,minGroups=%s," +
"degree=%s,capacity=%s,limit=%s," +
"maxDegree=%s,capacity=%s,limit=%s," +
"withIntermediary=%s,withVertex=%s}",
this.sources, this.label, this.direction,
this.minNeighbors, this.alpha,
this.minSimilars, this.top,
this.groupProperty, this.minGroups,
this.degree, this.capacity, this.limit,
this.maxDegree, this.capacity, this.limit,
this.withIntermediary, this.withVertex);
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_LIMIT;
import java.util.Map;
@ -71,10 +71,10 @@ public class JaccardSimilarityAPI extends TraverserAPI {
@QueryParam("direction") String direction,
@QueryParam("label") String edgeLabel,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree) {
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree) {
LOG.debug("Graph [{}] get jaccard similarity between '{}' and '{}' " +
"with direction {}, edge label {} and max degree '{}'",
graph, vertex, other, direction, edgeLabel, degree);
graph, vertex, other, direction, edgeLabel, maxDegree);
Id sourceId = VertexAPI.checkAndParseVertexId(vertex);
Id targetId = VertexAPI.checkAndParseVertexId(other);
@ -84,8 +84,8 @@ public class JaccardSimilarityAPI extends TraverserAPI {
double similarity;
try (JaccardSimilarTraverser traverser =
new JaccardSimilarTraverser(g)) {
similarity = traverser.jaccardSimilarity(sourceId, targetId,
dir, edgeLabel, degree);
similarity = traverser.jaccardSimilarity(sourceId, targetId, dir,
edgeLabel, maxDegree);
}
return JsonUtil.toJson(ImmutableMap.of("jaccard_similarity",
similarity));
@ -132,9 +132,9 @@ public class JaccardSimilarityAPI extends TraverserAPI {
@JsonProperty("step")
public TraverserAPI.Step step;
@JsonProperty("top")
public int top = Integer.valueOf(DEFAULT_LIMIT);
public int top = Integer.parseInt(DEFAULT_LIMIT);
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@Override
public String toString() {

View File

@ -19,9 +19,8 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_ELEMENTS_LIMIT;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import java.util.ArrayList;
import java.util.HashSet;
@ -76,14 +75,14 @@ public class KneighborAPI extends TraverserAPI {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("limit")
@DefaultValue(DEFAULT_ELEMENTS_LIMIT) long limit) {
LOG.debug("Graph [{}] get k-neighbor from '{}' with " +
"direction '{}', edge label '{}', max depth '{}', " +
"max degree '{}' and limit '{}'",
graph, sourceV, direction, edgeLabel, depth,
degree, limit);
maxDegree, limit);
Id source = VertexAPI.checkAndParseVertexId(sourceV);
Directions dir = Directions.convert(EdgeAPI.parseDirection(direction));
@ -93,7 +92,7 @@ public class KneighborAPI extends TraverserAPI {
Set<Id> ids;
try (KneighborTraverser traverser = new KneighborTraverser(g)) {
ids = traverser.kneighbor(source, dir, edgeLabel,
depth, degree, limit);
depth, maxDegree, limit);
}
return manager.serializer(g).writeList("vertices", ids);
}
@ -171,7 +170,7 @@ public class KneighborAPI extends TraverserAPI {
@JsonProperty("max_depth")
public int maxDepth;
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_ELEMENTS_LIMIT);
@JsonProperty("count_only")
public boolean countOnly = false;
@JsonProperty("with_vertex")

View File

@ -20,9 +20,8 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_ELEMENTS_LIMIT;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import java.util.ArrayList;
import java.util.HashSet;
@ -80,7 +79,7 @@ public class KoutAPI extends TraverserAPI {
@QueryParam("nearest")
@DefaultValue("true") boolean nearest,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity,
@QueryParam("limit")
@ -89,7 +88,7 @@ public class KoutAPI extends TraverserAPI {
"direction '{}', edge label '{}', max depth '{}', nearest " +
"'{}', max degree '{}', capacity '{}' and limit '{}'",
graph, source, direction, edgeLabel, depth, nearest,
degree, capacity, limit);
maxDegree, capacity, limit);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Directions dir = Directions.convert(EdgeAPI.parseDirection(direction));
@ -99,7 +98,7 @@ public class KoutAPI extends TraverserAPI {
Set<Id> ids;
try (KoutTraverser traverser = new KoutTraverser(g)) {
ids = traverser.kout(sourceId, dir, edgeLabel, depth,
nearest, degree, capacity, limit);
nearest, maxDegree, capacity, limit);
}
return manager.serializer(g).writeList("vertices", ids);
}
@ -185,9 +184,9 @@ public class KoutAPI extends TraverserAPI {
@JsonProperty("count_only")
public boolean countOnly = false;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_ELEMENTS_LIMIT);
@JsonProperty("with_vertex")
public boolean withVertex = false;
@JsonProperty("with_path")

View File

@ -112,7 +112,7 @@ public class MultiNodeShortestPathAPI extends TraverserAPI {
@JsonProperty("max_depth")
public int maxDepth;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("with_vertex")
public boolean withVertex = false;

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEPTH;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
@ -48,6 +48,7 @@ import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.codahale.metrics.annotation.Timed;
import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty;
@Path("graphs/{graph}/traversers/neighborrank")
@ -123,8 +124,9 @@ public class NeighborRankAPI extends API {
public Directions direction;
@JsonProperty("labels")
public List<String> labels;
@JsonProperty("degree")
public long degree = Long.parseLong(DEFAULT_DEGREE);
@JsonAlias("degree")
@JsonProperty("max_degree")
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("skip_degree")
public long skipDegree = 0L;
@JsonProperty("top")
@ -134,15 +136,15 @@ public class NeighborRankAPI extends API {
@Override
public String toString() {
return String.format("Step{direction=%s,labels=%s,degree=%s," +
return String.format("Step{direction=%s,labels=%s,maxDegree=%s," +
"top=%s}", this.direction, this.labels,
this.degree, this.top);
this.maxDegree, this.top);
}
private NeighborRankTraverser.Step jsonToStep(HugeGraph g) {
return new NeighborRankTraverser.Step(g, this.direction,
this.labels,
this.degree,
this.maxDegree,
this.skipDegree,
this.top,
DEFAULT_CAPACITY_PER_LAYER);

View File

@ -25,7 +25,7 @@ import java.util.Iterator;
import java.util.Set;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import javax.inject.Singleton;
@ -76,7 +76,7 @@ public class PathsAPI extends TraverserAPI {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity,
@QueryParam("limit")
@ -85,7 +85,7 @@ public class PathsAPI extends TraverserAPI {
"direction {}, edge label {}, max depth '{}', " +
"max degree '{}', capacity '{}' and limit '{}'",
graph, source, target, direction, edgeLabel, depth,
degree, capacity, limit);
maxDegree, capacity, limit);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Id targetId = VertexAPI.checkAndParseVertexId(target);
@ -95,7 +95,7 @@ public class PathsAPI extends TraverserAPI {
PathsTraverser traverser = new PathsTraverser(g);
HugeTraverser.PathSet paths = traverser.paths(sourceId, dir, targetId,
dir.opposite(), edgeLabel,
depth, degree, capacity,
depth, maxDegree, capacity,
limit);
return manager.serializer(g).writePaths("paths", paths, false);
}
@ -164,9 +164,9 @@ public class PathsAPI extends TraverserAPI {
@JsonProperty("nearest")
public boolean nearest = false;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_PATHS_LIMIT);
@JsonProperty("with_vertex")
public boolean withVertex = false;

View File

@ -43,7 +43,7 @@ import com.baidu.hugegraph.util.Log;
import com.codahale.metrics.annotation.Timed;
import com.fasterxml.jackson.annotation.JsonProperty;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_LIMIT;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEPTH;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.NO_LIMIT;
@ -68,29 +68,29 @@ public class PersonalRankAPI extends API {
E.checkArgument(request.alpha > 0 && request.alpha <= 1.0,
"The alpha of rank request must be in range (0, 1], " +
"but got '%s'", request.alpha);
E.checkArgument(request.degree > 0 || request.degree == NO_LIMIT,
"The degree of rank request must be > 0, but got: %s",
request.degree);
E.checkArgument(request.limit > 0 || request.limit == NO_LIMIT,
"The limit of rank request must be > 0, but got: %s",
request.limit);
E.checkArgument(request.maxDepth > 0 &&
request.maxDepth <= Long.valueOf(DEFAULT_MAX_DEPTH),
E.checkArgument(request.maxDegree > 0L || request.maxDegree == NO_LIMIT,
"The max degree of rank request must be > 0 " +
"or == -1, but got: %s", request.maxDegree);
E.checkArgument(request.limit > 0L || request.limit == NO_LIMIT,
"The limit of rank request must be > 0 or == -1, " +
"but got: %s", request.limit);
E.checkArgument(request.maxDepth > 0L &&
request.maxDepth <= Long.parseLong(DEFAULT_MAX_DEPTH),
"The max depth of rank request must be " +
"in range (0, %s], but got '%s'",
DEFAULT_MAX_DEPTH, request.maxDepth);
LOG.debug("Graph [{}] get personal rank from '{}' with " +
"edge label '{}', alpha '{}', degree '{}', " +
"edge label '{}', alpha '{}', maxDegree '{}', " +
"max depth '{}' and sorted '{}'",
graph, request.source, request.label, request.alpha,
request.degree, request.maxDepth, request.sorted);
request.maxDegree, request.maxDepth, request.sorted);
Id sourceId = HugeVertex.getIdValue(request.source);
HugeGraph g = graph(manager, graph);
PersonalRankTraverser traverser;
traverser = new PersonalRankTraverser(g, request.alpha, request.degree,
traverser = new PersonalRankTraverser(g, request.alpha, request.maxDegree,
request.maxDepth);
Map<Id, Double> ranks = traverser.personalRank(sourceId, request.label,
request.withLabel);
@ -106,10 +106,10 @@ public class PersonalRankAPI extends API {
private String label;
@JsonProperty("alpha")
private double alpha;
@JsonProperty("degree")
private long degree = Long.valueOf(DEFAULT_DEGREE);
@JsonProperty("max_degree")
private long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("limit")
private long limit = Long.valueOf(DEFAULT_LIMIT);
private long limit = Long.parseLong(DEFAULT_LIMIT);
@JsonProperty("max_depth")
private int maxDepth;
@JsonProperty("with_label")
@ -121,10 +121,10 @@ public class PersonalRankAPI extends API {
@Override
public String toString() {
return String.format("RankRequest{source=%s,label=%s,alpha=%s," +
"degree=%s,limit=%s,maxDepth=%s," +
"maxDegree=%s,limit=%s,maxDepth=%s," +
"withLabel=%s,sorted=%s}",
this.source, this.label, this.alpha,
this.degree, this.limit, this.maxDepth,
this.maxDegree, this.limit, this.maxDepth,
this.withLabel, this.sorted);
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import javax.inject.Singleton;
@ -63,7 +63,7 @@ public class RaysAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity,
@QueryParam("limit")
@ -71,7 +71,7 @@ public class RaysAPI extends API {
LOG.debug("Graph [{}] get rays paths from '{}' with " +
"direction '{}', edge label '{}', max depth '{}', " +
"max degree '{}', capacity '{}' and limit '{}'",
graph, sourceV, direction, edgeLabel, depth, degree,
graph, sourceV, direction, edgeLabel, depth, maxDegree,
capacity, limit);
Id source = VertexAPI.checkAndParseVertexId(sourceV);
@ -81,7 +81,7 @@ public class RaysAPI extends API {
SubGraphTraverser traverser = new SubGraphTraverser(g);
HugeTraverser.PathSet paths = traverser.rays(source, dir, edgeLabel,
depth, degree,
depth, maxDegree,
capacity, limit);
return manager.serializer(g).writePaths("rays", paths, false);
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import javax.inject.Singleton;
@ -65,7 +65,7 @@ public class RingsAPI extends API {
@QueryParam("source_in_ring")
@DefaultValue("true") boolean sourceInRing,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity,
@QueryParam("limit")
@ -75,7 +75,7 @@ public class RingsAPI extends API {
"source in ring '{}', max degree '{}', capacity '{}' " +
"and limit '{}'",
graph, sourceV, direction, edgeLabel, depth, sourceInRing,
degree, capacity, limit);
maxDegree, capacity, limit);
Id source = VertexAPI.checkAndParseVertexId(sourceV);
Directions dir = Directions.convert(EdgeAPI.parseDirection(direction));
@ -85,7 +85,7 @@ public class RingsAPI extends API {
SubGraphTraverser traverser = new SubGraphTraverser(g);
HugeTraverser.PathSet paths = traverser.rings(source, dir, edgeLabel,
depth, sourceInRing,
degree, capacity, limit);
maxDegree, capacity, limit);
return manager.serializer(g).writePaths("rings", paths, false);
}
}

View File

@ -19,7 +19,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import java.util.Set;
@ -63,12 +63,12 @@ public class SameNeighborsAPI extends API {
@QueryParam("direction") String direction,
@QueryParam("label") String edgeLabel,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("limit")
@DefaultValue(DEFAULT_PATHS_LIMIT) long limit) {
LOG.debug("Graph [{}] get same neighbors between '{}' and '{}' with " +
"direction {}, edge label {}, max degree '{}' and limit '{}'",
graph, vertex, other, direction, edgeLabel, degree, limit);
graph, vertex, other, direction, edgeLabel, maxDegree, limit);
Id sourceId = VertexAPI.checkAndParseVertexId(vertex);
Id targetId = VertexAPI.checkAndParseVertexId(other);
@ -77,7 +77,7 @@ public class SameNeighborsAPI extends API {
HugeGraph g = graph(manager, graph);
SameNeighborTraverser traverser = new SameNeighborTraverser(g);
Set<Id> neighbors = traverser.sameNeighbors(sourceId, targetId, dir,
edgeLabel, degree, limit);
edgeLabel, maxDegree, limit);
return manager.serializer(g).writeList("same_neighbors", neighbors);
}
}

View File

@ -22,7 +22,7 @@ package com.baidu.hugegraph.api.traversers;
import java.util.List;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import javax.inject.Singleton;
import javax.ws.rs.DefaultValue;
@ -66,16 +66,16 @@ public class ShortestPathAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("max_depth") int depth,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("skip_degree")
@DefaultValue("0") long skipDegree,
@QueryParam("capacity")
@DefaultValue(DEFAULT_CAPACITY) long capacity) {
LOG.debug("Graph [{}] get shortest path from '{}', to '{}' with " +
"direction {}, edge label {}, max depth '{}', " +
"max degree '{}', skipped degree '{}' and capacity '{}'",
"max degree '{}', skipped maxDegree '{}' and capacity '{}'",
graph, source, target, direction, edgeLabel, depth,
degree, skipDegree, capacity);
maxDegree, skipDegree, capacity);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Id targetId = VertexAPI.checkAndParseVertexId(target);
@ -89,7 +89,7 @@ public class ShortestPathAPI extends API {
ImmutableList.of(edgeLabel);
HugeTraverser.Path path = traverser.shortestPath(sourceId, targetId,
dir, edgeLabels, depth,
degree, skipDegree,
maxDegree, skipDegree,
capacity);
return manager.serializer(g).writeList("path", path.vertices());
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_PATHS_LIMIT;
import java.util.Iterator;
@ -67,7 +67,7 @@ public class SingleSourceShortestPathAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("weight") String weight,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("skip_degree")
@DefaultValue("0") long skipDegree,
@QueryParam("capacity")
@ -79,7 +79,7 @@ public class SingleSourceShortestPathAPI extends API {
"with direction {}, edge label {}, weight property {}, " +
"max degree '{}', limit '{}' and with vertex '{}'",
graph, source, direction, edgeLabel,
weight, degree, withVertex);
weight, maxDegree, withVertex);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Directions dir = Directions.convert(EdgeAPI.parseDirection(direction));
@ -89,7 +89,7 @@ public class SingleSourceShortestPathAPI extends API {
new SingleSourceShortestPathTraverser(g);
WeightedPaths paths = traverser.singleSourceShortestPaths(
sourceId, dir, edgeLabel, weight,
degree, skipDegree, capacity, limit);
maxDegree, skipDegree, capacity, limit);
Iterator<Vertex> iterator = QueryResults.emptyIterator();
assert paths != null;
if (!paths.isEmpty() && withVertex) {

View File

@ -117,7 +117,7 @@ public class TemplatePathsAPI extends TraverserAPI {
private static RepeatEdgeStep repeatEdgeStep(HugeGraph graph,
TemplatePathStep step) {
return new RepeatEdgeStep(graph, step.direction, step.labels,
step.properties, step.degree,
step.properties, step.maxDegree,
step.skipDegree, step.maxTimes);
}
@ -132,9 +132,9 @@ public class TemplatePathsAPI extends TraverserAPI {
@JsonProperty("with_ring")
public boolean withRing = false;
@JsonProperty("capacity")
public long capacity = Long.valueOf(DEFAULT_CAPACITY);
public long capacity = Long.parseLong(DEFAULT_CAPACITY);
@JsonProperty("limit")
public long limit = Long.valueOf(DEFAULT_PATHS_LIMIT);
public long limit = Long.parseLong(DEFAULT_PATHS_LIMIT);
@JsonProperty("with_vertex")
public boolean withVertex = false;
@ -157,10 +157,11 @@ public class TemplatePathsAPI extends TraverserAPI {
@Override
public String toString() {
return String.format("TemplatePathStep{direction=%s,labels=%s," +
"properties=%s,degree=%s,skipDegree=%s," +
"properties=%s,maxDegree=%s,skipDegree=%s," +
"maxTimes=%s}",
this.direction, this.labels, this.properties,
this.degree, this.skipDegree, this.maxTimes);
this.maxDegree, this.skipDegree,
this.maxTimes);
}
}
}

View File

@ -26,15 +26,16 @@ import com.baidu.hugegraph.HugeGraph;
import com.baidu.hugegraph.api.API;
import com.baidu.hugegraph.traversal.algorithm.steps.EdgeStep;
import com.baidu.hugegraph.type.define.Directions;
import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
public class TraverserAPI extends API {
protected static EdgeStep step(HugeGraph graph, Step step) {
return new EdgeStep(graph, step.direction, step.labels, step.properties,
step.degree, step.skipDegree);
step.maxDegree, step.skipDegree);
}
protected static class Step {
@ -45,17 +46,18 @@ public class TraverserAPI extends API {
public List<String> labels;
@JsonProperty("properties")
public Map<String, Object> properties;
@JsonProperty("degree")
public long degree = Long.valueOf(DEFAULT_DEGREE);
@JsonAlias("degree")
@JsonProperty("max_degree")
public long maxDegree = Long.parseLong(DEFAULT_MAX_DEGREE);
@JsonProperty("skip_degree")
public long skipDegree = 0L;
@Override
public String toString() {
return String.format("Step{direction=%s,labels=%s,properties=%s," +
"degree=%s,skipDegree=%s}",
"maxDegree=%s,skipDegree=%s}",
this.direction, this.labels, this.properties,
this.degree, this.skipDegree);
this.maxDegree, this.skipDegree);
}
}
}

View File

@ -20,7 +20,7 @@
package com.baidu.hugegraph.api.traversers;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_CAPACITY;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import java.util.Iterator;
@ -68,7 +68,7 @@ public class WeightedShortestPathAPI extends API {
@QueryParam("label") String edgeLabel,
@QueryParam("weight") String weight,
@QueryParam("max_degree")
@DefaultValue(DEFAULT_DEGREE) long degree,
@DefaultValue(DEFAULT_MAX_DEGREE) long maxDegree,
@QueryParam("skip_degree")
@DefaultValue("0") long skipDegree,
@QueryParam("capacity")
@ -78,8 +78,8 @@ public class WeightedShortestPathAPI extends API {
"'{}' with direction {}, edge label {}, weight property {}, " +
"max degree '{}', skip degree '{}', capacity '{}', " +
"and with vertex '{}'",
graph, source, target, direction, edgeLabel, weight, degree,
skipDegree, capacity, withVertex);
graph, source, target, direction, edgeLabel, weight,
maxDegree, skipDegree, capacity, withVertex);
Id sourceId = VertexAPI.checkAndParseVertexId(source);
Id targetId = VertexAPI.checkAndParseVertexId(target);
@ -92,7 +92,7 @@ public class WeightedShortestPathAPI extends API {
NodeWithWeight path = traverser.weightedShortestPath(
sourceId, targetId, dir, edgeLabel, weight,
degree, skipDegree, capacity);
maxDegree, skipDegree, capacity);
Iterator<Vertex> iterator = QueryResults.emptyIterator();
if (path != null && withVertex) {
assert !path.node().path().isEmpty();

View File

@ -20,6 +20,7 @@
package com.baidu.hugegraph.backend.store.cassandra;
import java.util.Iterator;
import java.util.List;
import java.util.function.BiFunction;
import com.baidu.hugegraph.backend.page.PageState;
@ -27,6 +28,7 @@ import com.baidu.hugegraph.backend.query.Query;
import com.baidu.hugegraph.backend.store.BackendEntry;
import com.baidu.hugegraph.backend.store.BackendEntryIterator;
import com.baidu.hugegraph.util.E;
import com.datastax.driver.core.ExecutionInfo;
import com.datastax.driver.core.PagingState;
import com.datastax.driver.core.ResultSet;
import com.datastax.driver.core.Row;
@ -37,7 +39,8 @@ public class CassandraEntryIterator extends BackendEntryIterator {
private final Iterator<Row> rows;
private final BiFunction<BackendEntry, Row, BackendEntry> merger;
private long remaining;
private int fetchdPageSize;
private long expected;
private BackendEntry next;
public CassandraEntryIterator(ResultSet results, Query query,
@ -45,16 +48,41 @@ public class CassandraEntryIterator extends BackendEntryIterator {
super(query);
this.results = results;
this.rows = results.iterator();
this.remaining = results.getAvailableWithoutFetching();
this.merger = merger;
this.fetchdPageSize = results.getAvailableWithoutFetching();
this.next = null;
this.skipOffset();
if (query.paging()) {
E.checkState(this.remaining == query.limit() ||
results.isFullyFetched(),
"Unexpected fetched page size: %s", this.remaining);
assert query.offset() == 0L;
assert query.limit() >= 0L || query.noLimit() : query.limit();
// Skip page offset
this.expected = PageState.fromString(query.page()).offset();
this.skipPageOffset(query.page());
// Check the number of available rows
E.checkState(this.fetchdPageSize <= query.limit(),
"Unexpected fetched page size: %s",
this.fetchdPageSize);
if (results.isFullyFetched()) {
/*
* All results fetched
* NOTE: it may be enough or not enough for the entire page
*/
this.expected = this.fetchdPageSize;
} else {
/*
* Not fully fetched, that's fetchdPageSize == query.limit(),
*
* NOTE: but there may be fetchdPageSize < query.limit(), means
* not fetched the entire page (ScyllaDB may go here #1340),
* try to fetch next page later until got the expected count.
* Can simulate by: `select.setFetchSize(total - 1)`
*/
this.expected = query.total();
}
} else {
this.expected = query.total();
this.skipOffset();
}
}
@ -71,11 +99,18 @@ public class CassandraEntryIterator extends BackendEntryIterator {
this.next = null;
}
while (this.remaining > 0 && this.rows.hasNext()) {
if (this.query.paging()) {
this.remaining--;
}
while (this.expected > 0L && this.rows.hasNext()) {
// Limit expected count, due to rows.hasNext() will fetch next page
this.expected--;
Row row = this.rows.next();
if (this.query.paging()) {
// Update fetchdPageSize if auto fetch the next page
if (this.expected > 0L && this.availableLocal() == 0) {
if (this.rows.hasNext()) {
this.fetchdPageSize = this.availableLocal();
}
}
}
BackendEntry merged = this.merger.apply(this.current, row);
if (this.current == null) {
// The first time to read
@ -112,11 +147,50 @@ public class CassandraEntryIterator extends BackendEntryIterator {
@Override
protected PageState pageState() {
PagingState page = this.results.getExecutionInfo().getPagingState();
if (page == null || this.results.isExhausted()) {
return new PageState(PageState.EMPTY_BYTES, 0, (int) this.count());
byte[] position;
int offset = 0;
int count = (int) this.count();
assert this.fetched() == count;
int extra = this.availableLocal();
List<ExecutionInfo> infos = this.results.getAllExecutionInfo();
if (extra > 0 && infos.size() >= 2) {
/*
* Go back to the previous page if there are still available
* results fetched to local memory but not consumed, and set page
* offset with consumed amount of results.
*
* Safely, we should get the remaining size of the current page by:
* `Whitebox.getInternalState(results, "currentPage").size()`
* instead of
* `results.getAvailableWithoutFetching()`
*/
ExecutionInfo previous = infos.get(infos.size() - 2);
PagingState page = previous.getPagingState();
position = page.toBytes();
offset = this.fetchdPageSize - extra;
} else {
PagingState page = this.results.getExecutionInfo().getPagingState();
if (page == null || this.expected > 0L) {
// Call isExhausted() will lead to try to fetch the next page
E.checkState(this.results.isExhausted(),
"Unexpected paging state with expected=%s, " +
"ensure consume all the fetched results before " +
"calling pageState()", this.expected);
position = PageState.EMPTY_BYTES;
} else {
/*
* Exist page position which used to fetch the next page.
* Maybe it happens to the last page (that's the position is
* at the end of results and next page is empty)
*/
position = page.toBytes();
}
}
byte[] position = page.toBytes();
return new PageState(position, 0, (int) this.count());
return new PageState(position, offset, count);
}
private int availableLocal() {
return this.results.getAvailableWithoutFetching();
}
}

View File

@ -204,18 +204,39 @@ public abstract class CassandraTable
}
protected void setPageState(Query query, List<Select> selects) {
if (query.noLimit()) {
if (query.noLimit() && !query.paging()) {
return;
}
for (Select select : selects) {
long total = query.total();
int total = (int) query.total();
if (!query.noLimit()) {
E.checkArgument(total == query.total(),
"Invalid query limit %s", query.limit());
} else {
assert total == -1 : total;
}
String page = query.page();
if (page == null) {
// Set limit
select.limit((int) total);
assert total > 0 : total;
select.limit(total);
} else {
select.setFetchSize((int) total);
// It's the first time if page is empty
/*
* NOTE: the `total` may be -1 when query.noLimit(),
* setFetchSize(-1) means the default fetch size will be used.
*/
assert total > 0 || total == -1 : total;
select.setFetchSize(total);
/*
* Can't set limit here `select.limit(total)`
* due to it will cause can't get the next page-state.
* Also can't set `select.limit(total + 1)` due to it will
* cause error "Paging state mismatch" when setPagingState().
*/
// It's the first time if page is empty, skip setPagingState
if (!page.isEmpty()) {
byte[] position = PageState.fromString(page).position();
try {

View File

@ -152,6 +152,11 @@
<artifactId>lz4-java</artifactId>
<version>1.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.20</version>
</dependency>
</dependencies>
<build>

View File

@ -106,16 +106,16 @@ public class StandardHugeGraph implements HugeGraph {
StandardHugeGraph.SysTransaction.class
};
public static final Set<ConfigOption> ALLOWED_CONFIGS = ImmutableSet.of(
CoreOptions.TASK_WAIT_TIMEOUT,
CoreOptions.TASK_SYNC_DELETION,
CoreOptions.TASK_TTL_DELETE_BATCH,
CoreOptions.TASK_INPUT_SIZE_LIMIT,
CoreOptions.TASK_RESULT_SIZE_LIMIT,
CoreOptions.OLTP_CONCURRENT_THREADS,
CoreOptions.OLTP_CONCURRENT_DEPTH,
CoreOptions.VERTEX_DEFAULT_LABEL,
CoreOptions.VERTEX_ENCODE_PK_NUMBER
public static final Set<ConfigOption<?>> ALLOWED_CONFIGS = ImmutableSet.of(
CoreOptions.TASK_WAIT_TIMEOUT,
CoreOptions.TASK_SYNC_DELETION,
CoreOptions.TASK_TTL_DELETE_BATCH,
CoreOptions.TASK_INPUT_SIZE_LIMIT,
CoreOptions.TASK_RESULT_SIZE_LIMIT,
CoreOptions.OLTP_CONCURRENT_THREADS,
CoreOptions.OLTP_CONCURRENT_DEPTH,
CoreOptions.VERTEX_DEFAULT_LABEL,
CoreOptions.VERTEX_ENCODE_PK_NUMBER
);
private static final Logger LOG = Log.logger(HugeGraph.class);

View File

@ -29,6 +29,8 @@ public class PageState {
public static final byte[] EMPTY_BYTES = new byte[0];
public static final PageState EMPTY = new PageState(EMPTY_BYTES, 0, 0);
public static final char SPACE = ' ';
public static final char PLUS = '+';
private final byte[] position;
private final int offset;
@ -72,6 +74,13 @@ public class PageState {
}
public static PageState fromString(String page) {
E.checkNotNull(page, "page");
/*
* URLDecoder will auto decode '+' to space in url due to the request
* of HTML4, so we choose to replace the space to '+' after getting it
* More details refer to #1437
*/
page = page.replace(SPACE, PLUS);
return fromBytes(toBytes(page));
}

View File

@ -46,10 +46,12 @@ public class BinaryEntryIterator<Elem> extends BackendEntryIterator {
this.merger = m;
this.next = null;
this.skipOffset();
if (query.paging()) {
assert query.offset() == 0L;
assert PageState.fromString(query.page()).offset() == 0;
this.skipPageOffset(query.page());
} else {
this.skipOffset();
}
}
@ -98,22 +100,9 @@ public class BinaryEntryIterator<Elem> extends BackendEntryIterator {
return this.current != null;
}
public final static long sizeOfBackendEntry(BackendEntry entry) {
/*
* 3 cases:
* 1) one vertex per entry
* 2) one edge per column (one entry <==> a vertex),
* 3) one element id per column (one entry <==> an index)
*/
if (entry.type().isEdge() || entry.type().isIndex()) {
return entry.columnsSize();
}
return 1L;
}
@Override
protected final long sizeOf(BackendEntry entry) {
return sizeOfBackendEntry(entry);
return sizeOfEntry(entry);
}
@Override
@ -140,10 +129,16 @@ public class BinaryEntryIterator<Elem> extends BackendEntryIterator {
((BinaryBackendEntry) this.current).removeColumn(lastOne);
}
private void skipPageOffset(String page) {
PageState pagestate = PageState.fromString(page);
if (pagestate.offset() > 0 && this.fetch()) {
this.skip(this.current, pagestate.offset());
public final static long sizeOfEntry(BackendEntry entry) {
/*
* 3 cases:
* 1) one vertex per entry
* 2) one edge per column (one entry <==> a vertex),
* 3) one element id per column (one entry <==> an index)
*/
if (entry.type().isEdge() || entry.type().isIndex()) {
return entry.columnsSize();
}
return 1L;
}
}

View File

@ -128,28 +128,47 @@ public abstract class BackendEntryIterator implements CIter<BackendEntry> {
return this.count + ccount;
}
protected final void skipPageOffset(String page) {
PageState pageState = PageState.fromString(page);
int pageOffset = pageState.offset();
if (pageOffset > 0) {
/*
* Don't update this.count even if skipped page offset,
* because the skipped records belongs to the last page.
*/
this.skipOffset(pageOffset);
}
}
protected void skipOffset() {
long offset = this.query.offset() - this.query.actualOffset();
if (offset <= 0L) {
return;
}
long skipped = this.skipOffset(offset);
this.count += skipped;
this.query.goOffset(skipped);
}
protected long skipOffset(long offset) {
assert offset >= 0L;
long skipped = 0L;
// Skip offset
while (this.count < offset && this.fetch()) {
while (skipped < offset && this.fetch()) {
assert this.current != null;
final long size = this.sizeOf(this.current);
this.count += size;
if (this.count > offset) {
skipped += size;
if (skipped > offset) {
// Skip part of sub-items in an entry
final long skip = size - (this.count - offset);
this.count -= this.skip(this.current, skip);
assert this.count == offset;
final long skip = size - (skipped - offset);
skipped -= this.skip(this.current, skip);
assert skipped == offset;
} else {
// Skip entry
this.current = null;
}
}
this.query.goOffset(this.count);
return skipped;
}
protected long sizeOf(BackendEntry entry) {

View File

@ -168,9 +168,9 @@ public abstract class BackendTable<Session extends BackendSession, Entry> {
long maxKey = this.maxKey();
double each = maxKey / count;
long offset = 0L;
String last = this.position(offset);
List<Shard> splits = new ArrayList<>((int) count);
String last = START;
long offset = 0L;
while (offset < maxKey) {
offset += each;
if (offset > maxKey) {

View File

@ -72,7 +72,7 @@ public final class RaftSharedContext {
public static final int BUSY_SLEEP_FACTOR = 3 * 1000;
public static final int WAIT_RPC_TIMEOUT = 30 * 60 * 1000;
// compress block size
public static final int BLOCK_SIZE = 4096;
public static final int BLOCK_SIZE = 8192;
public static final String DEFAULT_GROUP = "default";

View File

@ -37,9 +37,9 @@ import com.alipay.sofa.jraft.error.RaftError;
import com.alipay.sofa.jraft.storage.snapshot.SnapshotReader;
import com.alipay.sofa.jraft.storage.snapshot.SnapshotWriter;
import com.alipay.sofa.jraft.util.CRC64;
import com.baidu.hugegraph.util.CompressUtil;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.baidu.hugegraph.util.ZipUtil;
public class StoreSnapshotFile {
@ -133,7 +133,8 @@ public class StoreSnapshotFile {
String outputFile = Paths.get(writerPath, SNAPSHOT_ARCHIVE).toString();
try {
Checksum checksum = new CRC64();
ZipUtil.compress(writerPath, SNAPSHOT_DIR, outputFile, checksum);
CompressUtil.compressTar(writerPath, SNAPSHOT_DIR,
outputFile, checksum);
metaBuilder.setChecksum(Long.toHexString(checksum.getValue()));
if (writer.addFile(SNAPSHOT_ARCHIVE, metaBuilder.build())) {
done.run(Status.OK());
@ -155,7 +156,7 @@ public class StoreSnapshotFile {
throws IOException {
String sourceFile = Paths.get(readerPath, SNAPSHOT_ARCHIVE).toString();
Checksum checksum = new CRC64();
ZipUtil.decompress(sourceFile, readerPath, checksum);
CompressUtil.decompressTar(sourceFile, readerPath, checksum);
if (meta.hasChecksum()) {
E.checkArgument(meta.getChecksum().equals(
Long.toHexString(checksum.getValue())),

View File

@ -24,10 +24,15 @@ import java.io.IOException;
import java.util.Date;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.tinkerpop.gremlin.process.traversal.Path;
import org.apache.tinkerpop.gremlin.process.traversal.step.util.Tree;
import org.apache.tinkerpop.gremlin.structure.Element;
import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONIo;
import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTokens;
import org.apache.tinkerpop.gremlin.structure.io.graphson.TinkerPopJacksonModule;
import org.apache.tinkerpop.shaded.jackson.core.JsonGenerator;
import org.apache.tinkerpop.shaded.jackson.core.JsonParser;
@ -70,10 +75,10 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
private static final long serialVersionUID = 6480426922914059122L;
public static boolean OPTIMIZE_SERIALIZE = true;
private static final String TYPE_NAMESPACE = "hugegraph";
private static boolean OPTIMIZE_SERIALIZE = true;
@SuppressWarnings("rawtypes")
private static final Map<Class, String> TYPE_DEFINITIONS;
@ -198,6 +203,9 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
*/
module.addSerializer(HugeVertex.class, new HugeVertexSerializer());
module.addSerializer(HugeEdge.class, new HugeEdgeSerializer());
module.addSerializer(Path.class, new PathSerializer());
module.addSerializer(Tree.class, new TreeSerializer());
}
@SuppressWarnings("rawtypes")
@ -485,6 +493,49 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
}
}
private static class PathSerializer extends StdSerializer<Path> {
public PathSerializer() {
super(Path.class);
}
@Override
public void serialize(Path path, JsonGenerator jsonGenerator,
SerializerProvider provider) throws IOException {
jsonGenerator.writeStartObject();
jsonGenerator.writeObjectField(GraphSONTokens.LABELS,
path.labels());
jsonGenerator.writeObjectField(GraphSONTokens.OBJECTS,
path.objects());
jsonGenerator.writeEndObject();
}
}
@SuppressWarnings("rawtypes") // Tree<T>
private static class TreeSerializer extends StdSerializer<Tree> {
public TreeSerializer() {
super(Tree.class);
}
@Override
public void serialize(Tree tree, JsonGenerator jsonGenerator,
SerializerProvider provider) throws IOException {
jsonGenerator.writeStartArray();
@SuppressWarnings("unchecked")
Set<Map.Entry<Element, Tree>> set = tree.entrySet();
for (Map.Entry<Element, Tree> entry : set) {
jsonGenerator.writeStartObject();
jsonGenerator.writeObjectField(GraphSONTokens.KEY,
entry.getKey());
jsonGenerator.writeObjectField(GraphSONTokens.VALUE,
entry.getValue());
jsonGenerator.writeEndObject();
}
jsonGenerator.writeEndArray();
}
}
private static class ShardSerializer extends StdSerializer<Shard> {
public ShardSerializer() {
@ -493,8 +544,7 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
@Override
public void serialize(Shard shard, JsonGenerator jsonGenerator,
SerializerProvider provider)
throws IOException {
SerializerProvider provider) throws IOException {
jsonGenerator.writeStartObject();
jsonGenerator.writeStringField("start", shard.start());
jsonGenerator.writeStringField("end", shard.end());
@ -511,8 +561,7 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
@Override
public void serialize(File file, JsonGenerator jsonGenerator,
SerializerProvider provider)
throws IOException {
SerializerProvider provider) throws IOException {
jsonGenerator.writeStartObject();
jsonGenerator.writeStringField("file", file.getName());
jsonGenerator.writeEndObject();
@ -527,8 +576,7 @@ public class HugeGraphSONModule extends TinkerPopJacksonModule {
@Override
public void serialize(Blob blob, JsonGenerator jsonGenerator,
SerializerProvider provider)
throws IOException {
SerializerProvider provider) throws IOException {
jsonGenerator.writeBinary(blob.bytes());
}
}

View File

@ -26,7 +26,6 @@ import com.baidu.hugegraph.HugeGraph;
import com.baidu.hugegraph.backend.id.Id;
import com.baidu.hugegraph.backend.id.IdGenerator;
import com.baidu.hugegraph.backend.tx.SchemaTransaction;
import com.baidu.hugegraph.config.CoreOptions;
import com.baidu.hugegraph.exception.ExistedException;
import com.baidu.hugegraph.schema.EdgeLabel;
import com.baidu.hugegraph.schema.IndexLabel;

View File

@ -466,11 +466,15 @@ public class EdgeLabelBuilder extends AbstractBuilder
"must belong to the origin/new properties: %s/%s ",
this.nullableKeys, originProps, appendProps);
E.checkArgument(!CollectionUtil.hasIntersection(this.sortKeys,
List<String> sortKeys = edgeLabel == null ?
this.sortKeys :
this.graph()
.mapPkId2Name(edgeLabel.sortKeys());
E.checkArgument(!CollectionUtil.hasIntersection(sortKeys,
this.nullableKeys),
"The nullableKeys: %s are not allowed to " +
"belong to sortKeys: %s of edge label '%s'",
this.nullableKeys, this.sortKeys, this.name);
this.nullableKeys, sortKeys, this.name);
if (action == Action.APPEND) {
Collection<String> newAddedProps = CollectionUtils.subtract(

View File

@ -461,11 +461,15 @@ public class VertexLabelBuilder extends AbstractBuilder
"must belong to the origin/new properties: %s/%s",
this.nullableKeys, originProps, appendProps);
E.checkArgument(!CollectionUtil.hasIntersection(this.primaryKeys,
List<String> primaryKeys = vertexLabel == null ?
this.primaryKeys :
this.graph()
.mapPkId2Name(vertexLabel.primaryKeys());
E.checkArgument(!CollectionUtil.hasIntersection(primaryKeys,
this.nullableKeys),
"The nullableKeys: %s are not allowed to " +
"belong to primaryKeys: %s of vertex label '%s'",
this.nullableKeys, this.primaryKeys, this.name);
this.nullableKeys, primaryKeys, this.name);
if (action == Action.APPEND) {
Collection<String> newAddedProps = CollectionUtils.subtract(

View File

@ -108,6 +108,11 @@ public class HugeSecurityManager extends SecurityManager {
"com.baidu.hugegraph.backend.store.raft.rpc.RpcForwarder"
);
private static final Map<String, Set<String>> NEW_SECURITY_EXCEPTION = ImmutableMap.of(
"com.baidu.hugegraph.security.HugeSecurityManager",
ImmutableSet.of("newSecurityException")
);
@Override
public void checkPermission(Permission permission) {
if (DENIED_PERMISSIONS.contains(permission.getName()) &&
@ -306,7 +311,7 @@ public class HugeSecurityManager extends SecurityManager {
@Override
public void checkPropertiesAccess() {
if (callFromGremlin()) {
if (callFromGremlin() && !callFromNewSecurityException()) {
throw newSecurityException(
"Not allowed to access system properties via Gremlin");
}
@ -441,6 +446,10 @@ public class HugeSecurityManager extends SecurityManager {
return callFromWorkerWithClass(RAFT_CLASSES);
}
private static boolean callFromNewSecurityException() {
return callFromMethods(NEW_SECURITY_EXCEPTION);
}
private static boolean callFromWorkerWithClass(Set<String> classes) {
Thread curThread = Thread.currentThread();
if (curThread.getName().startsWith(GREMLIN_SERVER_WORKER) ||

View File

@ -222,9 +222,13 @@ public class HugeTask<V> extends FutureTask<V> {
return this.result;
}
private void result(String result) {
private synchronized boolean result(TaskStatus status, String result) {
checkPropertySize(result, P.RESULT);
this.result = result;
if (this.status(status)) {
this.result = result;
return true;
}
return false;
}
public void server(Id server) {
@ -319,18 +323,17 @@ public class HugeTask<V> extends FutureTask<V> {
LOG.warn("An exception occurred when running task: {}",
this.id(), e);
// Update status to FAILED if exception occurred(not interrupted)
if (this.status(TaskStatus.FAILED)) {
this.result(e.toString());
if (this.result(TaskStatus.FAILED, e.toString())) {
return true;
}
}
return false;
}
public void failSave(Throwable e) {
public void failToSave(Throwable e) {
if (!this.fail(e)) {
// Can't update status, just set result to error message
this.result(e.toString());
this.result = e.toString();
}
}
@ -352,9 +355,7 @@ public class HugeTask<V> extends FutureTask<V> {
protected void set(V v) {
String result = JsonUtil.toJson(v);
checkPropertySize(result, P.RESULT);
if (this.status(TaskStatus.SUCCESS)) {
this.result = result;
} else {
if (!this.result(TaskStatus.SUCCESS, result)) {
assert this.completed();
}
// Will call done() and may cause to save to store
@ -381,22 +382,21 @@ public class HugeTask<V> extends FutureTask<V> {
if (this.dependencies == null || this.dependencies.isEmpty()) {
return true;
}
TaskScheduler scheduler = this.scheduler();
for (Id dependency : this.dependencies) {
HugeTask<?> task = this.scheduler().task(dependency);
HugeTask<?> task = scheduler.task(dependency);
if (!task.completed()) {
// Dependent task not completed, re-schedule self
this.scheduler().schedule(this);
scheduler.schedule(this);
return false;
} else if (task.status() == TaskStatus.CANCELLED) {
this.status(TaskStatus.CANCELLED);
this.result(String.format(
this.result(TaskStatus.CANCELLED, String.format(
"Cancelled due to dependent task '%s' cancelled",
dependency));
this.done();
return false;
} else if (task.status() == TaskStatus.FAILED) {
this.status(TaskStatus.FAILED);
this.result(String.format(
this.result(TaskStatus.FAILED, String.format(
"Failed due to dependent task '%s' failed",
dependency));
this.done();
@ -483,7 +483,7 @@ public class HugeTask<V> extends FutureTask<V> {
}
}
protected Object[] asArray() {
protected synchronized Object[] asArray() {
E.checkState(this.type != null, "Task type can't be null");
E.checkState(this.name != null, "Task name can't be null");
@ -563,7 +563,7 @@ public class HugeTask<V> extends FutureTask<V> {
return this.asMap(true);
}
public Map<String, Object> asMap(boolean withDetails) {
public synchronized Map<String, Object> asMap(boolean withDetails) {
E.checkState(this.type != null, "Task type can't be null");
E.checkState(this.name != null, "Task name can't be null");

View File

@ -205,6 +205,15 @@ public class StandardTaskScheduler implements TaskScheduler {
public <V> Future<?> schedule(HugeTask<V> task) {
E.checkArgumentNotNull(task, "Task can't be null");
if (task.status() == TaskStatus.QUEUED) {
/*
* Just submit to queue if status=QUEUED (means re-schedule task)
* NOTE: schedule() method may be called multi times by
* HugeTask.checkDependenciesSuccess() method
*/
return this.resubmitTask(task);
}
if (task.callable() instanceof EphemeralJob) {
/*
* Due to EphemeralJob won't be serialized and deserialized through
@ -252,6 +261,16 @@ public class StandardTaskScheduler implements TaskScheduler {
return this.taskExecutor.submit(task);
}
private <V> Future<?> resubmitTask(HugeTask<V> task) {
E.checkArgument(task.status() == TaskStatus.QUEUED,
"Can't resubmit task '%s' with status %s",
task.id(), TaskStatus.QUEUED);
E.checkArgument(this.tasks.containsKey(task.id()),
"Can't resubmit task '%s' not been submitted before",
task.id());
return this.taskExecutor.submit(task);
}
public <V> void initTaskCallable(HugeTask<V> task) {
task.scheduler(this);

View File

@ -20,6 +20,7 @@
package com.baidu.hugegraph.task;
import java.util.Date;
import java.util.Set;
import java.util.concurrent.Callable;
import org.apache.tinkerpop.gremlin.structure.Transaction;
@ -30,13 +31,29 @@ import com.baidu.hugegraph.HugeGraph;
import com.baidu.hugegraph.HugeGraphParams;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.Log;
import com.google.common.collect.ImmutableSet;
public abstract class TaskCallable<V> implements Callable<V> {
private static final Logger LOG = Log.logger(HugeTask.class);
private static final String ERROR_MAX_LEN = "Failed to commit changes: " +
"The max length of bytes is";
private static final String ERROR_COMMIT = "Failed to commit changes: ";
private static final Set<String> ERROR_MESSAGES = ImmutableSet.of(
/*
* "The max length of bytes is" exception message occurs when
* task input size exceeds TASK_INPUT_SIZE_LIMIT or task result size
* exceeds TASK_RESULT_SIZE_LIMIT
*/
"The max length of bytes is",
/*
* "Batch too large" exception message occurs when using
* cassandra store and task input size is in
* [batch_size_fail_threshold_in_kb, TASK_INPUT_SIZE_LIMIT) or
* task result size is in
* [batch_size_fail_threshold_in_kb, TASK_RESULT_SIZE_LIMIT)
*/
"Batch too large"
);
private HugeTask<V> task = null;
private HugeGraph graph = null;
@ -65,7 +82,7 @@ public abstract class TaskCallable<V> implements Callable<V> {
public void setMinSaveInterval(long seconds) {
E.checkArgument(seconds > 0,
"Must set interval > 0, bug got '%s'", seconds);
"Must set interval > 0, but got '%s'", seconds);
this.saveInterval = seconds * 1000L;
}
@ -98,8 +115,9 @@ public abstract class TaskCallable<V> implements Callable<V> {
*/
LOG.error("Failed to save task with error \"{}\": {}",
e, task.asMap(false));
if (e.getMessage().contains(ERROR_MAX_LEN)) {
task.failSave(e);
String message = e.getMessage();
if (message.contains(ERROR_COMMIT) && needSaveWithEx(message)) {
task.failToSave(e);
this.graph().taskScheduler().save(task);
return;
}
@ -138,6 +156,15 @@ public abstract class TaskCallable<V> implements Callable<V> {
}
}
private static boolean needSaveWithEx(String message) {
for (String error : ERROR_MESSAGES) {
if (message.contains(error)) {
return true;
}
}
return false;
}
public static <V> TaskCallable<V> empty(Exception e) {
return new TaskCallable<V>() {
@Override

View File

@ -44,6 +44,7 @@ import com.baidu.hugegraph.backend.query.ConditionQuery;
import com.baidu.hugegraph.backend.query.Query;
import com.baidu.hugegraph.backend.query.QueryResults;
import com.baidu.hugegraph.backend.tx.GraphTransaction;
import com.baidu.hugegraph.backend.tx.GraphTransaction.LimitIterator;
import com.baidu.hugegraph.config.CoreOptions;
import com.baidu.hugegraph.exception.NotFoundException;
import com.baidu.hugegraph.iterator.ExtendableIterator;
@ -71,7 +72,7 @@ public class HugeTraverser {
public static final String DEFAULT_ELEMENTS_LIMIT = "10000000";
public static final String DEFAULT_PATHS_LIMIT = "10";
public static final String DEFAULT_LIMIT = "100";
public static final String DEFAULT_DEGREE = "10000";
public static final String DEFAULT_MAX_DEGREE = "10000";
public static final String DEFAULT_SKIP_DEGREE = "100000";
public static final String DEFAULT_SAMPLE = "100";
public static final String DEFAULT_MAX_DEPTH = "50";
@ -96,9 +97,10 @@ public class HugeTraverser {
return this.graph.option(CoreOptions.OLTP_CONCURRENT_DEPTH);
}
protected Set<Id> adjacentVertices(Set<Id> vertices, Directions dir,
Id label, Set<Id> excluded,
long degree, long limit) {
protected Set<Id> adjacentVertices(Id sourceV, Set<Id> vertices,
Directions dir, Id label,
Set<Id> excluded, long degree,
long limit) {
if (limit == 0) {
return ImmutableSet.of();
}
@ -110,7 +112,10 @@ public class HugeTraverser {
while (edges.hasNext()) {
HugeEdge e = (HugeEdge) edges.next();
Id target = e.id().otherVertexId();
if (excluded != null && excluded.contains(target)) {
boolean matchExcluded = (excluded != null &&
excluded.contains(target));
if (matchExcluded || neighbors.contains(target) ||
sourceV.equals(target)) {
continue;
}
neighbors.add(target);
@ -140,15 +145,19 @@ public class HugeTraverser {
return neighbors;
}
protected Set<Node> adjacentVertices(Set<Node> vertices, EdgeStep step,
Set<Node> excluded, long remaining) {
protected Set<Node> adjacentVertices(Id start, Set<Node> vertices,
EdgeStep step, Set<Node> excluded,
long remaining) {
Set<Node> neighbors = newSet();
for (Node source : vertices) {
Iterator<Edge> edges = this.edgesOfVertex(source.id(), step);
while (edges.hasNext()) {
Id target = ((HugeEdge) edges.next()).id().otherVertexId();
KNode kNode = new KNode(target, (KNode) source);
if (excluded != null && excluded.contains(kNode)) {
boolean matchExcluded = (excluded != null &&
excluded.contains(kNode));
if (matchExcluded || neighbors.contains(kNode) ||
start.equals(kNode.id())) {
continue;
}
neighbors.add(kNode);
@ -182,10 +191,17 @@ public class HugeTraverser {
ExtendableIterator<Edge> results = new ExtendableIterator<>();
for (Id label : labels.keySet()) {
E.checkNotNull(label, "edge label");
// TODO: limit should be applied to all labels
results.extend(this.edgesOfVertex(source, dir, label, limit));
}
return results;
if (limit == NO_LIMIT) {
return results;
}
long[] count = new long[1];
return new LimitIterator<>(results, e -> {
return count[0]++ >= limit;
});
}
protected Iterator<Edge> edgesOfVertex(Id source, EdgeStep edgeStep) {
@ -358,14 +374,14 @@ public class HugeTraverser {
Query.DEFAULT_CAPACITY, skipDegree);
if (capacity != NO_LIMIT) {
E.checkArgument(degree != NO_LIMIT && degree < capacity,
"The degree must be < capacity");
"The max degree must be < capacity");
E.checkArgument(skipDegree < capacity,
"The skipped degree must be < capacity");
}
if (skipDegree > 0L) {
E.checkArgument(degree != NO_LIMIT && skipDegree >= degree,
"The skipped degree must be >= degree, " +
"but got skipped degree '%s' and degree '%s'",
"The skipped degree must be >= max degree, " +
"but got skipped degree '%s' and max degree '%s'",
skipDegree, degree);
}
}

View File

@ -47,15 +47,14 @@ public class KneighborTraverser extends OltpTraverser {
Id labelId = this.getEdgeLabelId(label);
Set<Id> latest = newSet();
latest.add(sourceV);
Set<Id> all = newSet();
all.add(sourceV);
latest.add(sourceV);
while (depth-- > 0) {
long remaining = limit == NO_LIMIT ? NO_LIMIT : limit - all.size();
latest = this.adjacentVertices(latest, dir, labelId, all,
degree, remaining);
latest = this.adjacentVertices(sourceV, latest, dir, labelId,
all, degree, remaining);
all.addAll(latest);
if (limit != NO_LIMIT && all.size() >= limit) {
break;
@ -86,11 +85,10 @@ public class KneighborTraverser extends OltpTraverser {
Node sourceV = new KNode(source, null);
latest.add(sourceV);
all.add(sourceV);
while (maxDepth-- > 0) {
long remaining = limit == NO_LIMIT ? NO_LIMIT : limit - all.size();
latest = this.adjacentVertices(latest, step, all,
latest = this.adjacentVertices(source, latest, step, all,
remaining, single);
int size = all.size() + latest.size();
if (limit != NO_LIMIT && size >= limit) {

View File

@ -70,12 +70,12 @@ public class KoutTraverser extends OltpTraverser {
remaining = limit;
}
if (nearest) {
latest = this.adjacentVertices(latest, dir, labelId, all,
degree, remaining);
latest = this.adjacentVertices(sourceV, latest, dir, labelId,
all, degree, remaining);
all.addAll(latest);
} else {
latest = this.adjacentVertices(latest, dir, labelId, null,
degree, remaining);
latest = this.adjacentVertices(sourceV, latest, dir, labelId,
null, degree, remaining);
}
if (capacity != NO_LIMIT) {
// Update 'remaining' value to record remaining capacity
@ -130,11 +130,11 @@ public class KoutTraverser extends OltpTraverser {
NO_LIMIT : capacity - latest.size();
while (depth-- > 0) {
if (nearest) {
latest = this.adjacentVertices(latest, step, all,
latest = this.adjacentVertices(source, latest, step, all,
remaining, single);
all.addAll(latest);
} else {
latest = this.adjacentVertices(latest, step, null,
latest = this.adjacentVertices(source, latest, step, null,
remaining, single);
}
if (capacity != NO_LIMIT) {

View File

@ -36,8 +36,6 @@ import com.baidu.hugegraph.traversal.algorithm.steps.EdgeStep;
import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.E;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.Path.EMPTY_PATH;
public class MultiNodeShortestPathTraverser extends OltpTraverser {
public MultiNodeShortestPathTraverser(HugeGraph graph) {
@ -82,7 +80,7 @@ public class MultiNodeShortestPathTraverser extends OltpTraverser {
this.traversePairs(pairs.iterator(), pair -> {
Path path = traverser.shortestPath(pair.getLeft(), pair.getRight(),
step, maxDepth, capacity);
if (!EMPTY_PATH.equals(path)) {
if (!Path.EMPTY_PATH.equals(path)) {
results.add(path);
}
});
@ -99,7 +97,7 @@ public class MultiNodeShortestPathTraverser extends OltpTraverser {
for (Pair<Id, Id> pair : pairs) {
Path path = traverser.shortestPath(pair.getLeft(), pair.getRight(),
step, maxDepth, capacity);
if (!EMPTY_PATH.equals(path)) {
if (!Path.EMPTY_PATH.equals(path)) {
results.add(path);
}
}
@ -109,7 +107,7 @@ public class MultiNodeShortestPathTraverser extends OltpTraverser {
private static <T> void cmn(List<T> all, int m, int n, int current,
List<T> result, Consumer<List<T>> consumer) {
assert m <= all.size();
assert current < all.size();
assert current <= all.size();
if (result == null) {
result = new ArrayList<>(n);
}

View File

@ -244,7 +244,8 @@ public class NeighborRankTraverser extends HugeTraverser {
public Step(HugeGraph g, Directions direction, List<String> labels,
long degree, long skipDegree, int top, int capacity) {
E.checkArgument(top > 0 && top <= MAX_TOP,
"The top of each layer can't exceed %s", MAX_TOP);
"The top of each layer must be in (0, %s], but " +
"got %s", MAX_TOP, top);
E.checkArgument(capacity > 0,
"The capacity of each layer must be > 0, " +
"but got %s", capacity);
@ -281,7 +282,8 @@ public class NeighborRankTraverser extends HugeTraverser {
public long degree() {
E.checkArgument(this.degree > 0,
"The degree must be > 0, but got %s", this.degree);
"The max degree must be > 0, but got %s",
this.degree);
return this.degree;
}

View File

@ -81,11 +81,11 @@ public abstract class OltpTraverser extends HugeTraverser
}
}
protected Set<Node> adjacentVertices(Set<Node> latest, EdgeStep step,
Set<Node> all, long remaining,
boolean single) {
protected Set<Node> adjacentVertices(Id source, Set<Node> latest,
EdgeStep step, Set<Node> all,
long remaining, boolean single) {
if (single) {
return this.adjacentVertices(latest, step, all, remaining);
return this.adjacentVertices(source, latest, step, all, remaining);
} else {
AtomicLong remain = new AtomicLong(remaining);
return this.adjacentVertices(latest, step, all, remain);

View File

@ -60,7 +60,7 @@ public class PathsTraverser extends HugeTraverser {
PathSet paths = new PathSet();
if (sourceV.equals(targetV)) {
paths.add(new Path(sourceV, ImmutableList.of(sourceV)));
return paths;
}
Id labelId = this.getEdgeLabelId(label);

View File

@ -38,6 +38,7 @@ import com.baidu.hugegraph.type.define.Directions;
import com.baidu.hugegraph.util.CollectionUtil;
import com.baidu.hugegraph.util.E;
import com.baidu.hugegraph.util.InsertionOrderUtil;
import com.baidu.hugegraph.util.NumericUtil;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
@ -212,7 +213,8 @@ public class SingleSourceShortestPathTraverser extends HugeTraverser {
!edge.property(this.weight).isPresent()) {
edgeWeight = 1.0;
} else {
edgeWeight = edge.value(this.weight);
edgeWeight = NumericUtil.convertToNumber(
edge.value(this.weight)).doubleValue();
}
return edgeWeight;
}

View File

@ -19,7 +19,7 @@
package com.baidu.hugegraph.traversal.algorithm.steps;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.NO_LIMIT;
import java.util.HashMap;
@ -66,14 +66,14 @@ public class EdgeStep {
public EdgeStep(HugeGraph g, Directions direction, List<String> labels,
Map<String, Object> properties) {
this(g, direction, labels, properties,
Long.valueOf(DEFAULT_DEGREE), 0L);
Long.parseLong(DEFAULT_MAX_DEGREE), 0L);
}
public EdgeStep(HugeGraph g, Directions direction, List<String> labels,
Map<String, Object> properties,
long degree, long skipDegree) {
E.checkArgument(degree == NO_LIMIT || degree > 0L,
"The degree must be > 0 or == -1, but got: %s",
"The max degree must be > 0 or == -1, but got: %s",
degree);
HugeTraverser.checkSkipDegree(skipDegree, degree,
HugeTraverser.NO_LIMIT);

View File

@ -19,7 +19,7 @@
package com.baidu.hugegraph.traversal.algorithm.steps;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import java.util.List;
import java.util.Map;
@ -76,7 +76,7 @@ public class RepeatEdgeStep extends EdgeStep {
List<String> labels,
Map<String, Object> properties) {
this(g, direction, labels, properties,
Long.valueOf(DEFAULT_DEGREE), 0L, 1);
Long.parseLong(DEFAULT_MAX_DEGREE), 0L, 1);
}
public RepeatEdgeStep(HugeGraph g, Directions direction,

View File

@ -19,7 +19,7 @@
package com.baidu.hugegraph.traversal.algorithm.steps;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_MAX_DEGREE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_SAMPLE;
import static com.baidu.hugegraph.traversal.algorithm.HugeTraverser.NO_LIMIT;
@ -59,24 +59,24 @@ public class WeightedEdgeStep {
public WeightedEdgeStep(HugeGraph g, Directions direction, List<String> labels,
Map<String, Object> properties) {
this(g, direction, labels, properties,
Long.valueOf(DEFAULT_DEGREE), 0L, null, 0.0D,
Long.valueOf(DEFAULT_SAMPLE));
Long.parseLong(DEFAULT_MAX_DEGREE), 0L, null, 0.0D,
Long.parseLong(DEFAULT_SAMPLE));
}
public WeightedEdgeStep(HugeGraph g, Directions direction, List<String> labels,
Map<String, Object> properties,
long degree, long skipDegree,
long maxDegree, long skipDegree,
String weightBy, double defaultWeight, long sample) {
E.checkArgument(sample > 0L || sample == NO_LIMIT,
"The sample must be > 0 or == -1, but got: %s",
sample);
E.checkArgument(degree == NO_LIMIT || degree >= sample,
"Degree must be greater than or equal to sample," +
" but got degree %s and sample %s",
degree, sample);
E.checkArgument(maxDegree == NO_LIMIT || maxDegree >= sample,
"The max degree must be greater than or equal to " +
"sample, but got max degree %s and sample %s",
maxDegree, sample);
this.edgeStep = new EdgeStep(g, direction, labels, properties,
degree, skipDegree);
maxDegree, skipDegree);
if (weightBy != null) {
this.weightBy = g.propertyKey(weightBy);
} else {

View File

@ -0,0 +1,241 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.baidu.hugegraph.util;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.zip.CheckedInputStream;
import java.util.zip.CheckedOutputStream;
import java.util.zip.Checksum;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
import org.apache.commons.compress.archivers.ArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.io.output.NullOutputStream;
import com.baidu.hugegraph.backend.store.raft.RaftSharedContext;
import net.jpountz.lz4.LZ4BlockInputStream;
import net.jpountz.lz4.LZ4BlockOutputStream;
import net.jpountz.lz4.LZ4Compressor;
import net.jpountz.lz4.LZ4Factory;
import net.jpountz.lz4.LZ4FastDecompressor;
public final class CompressUtil {
/**
* Reference: https://mkyong.com/java/how-to-create-tar-gz-in-java/
*/
public static void compressTar(String rootDir, String sourceDir,
String outputFile, Checksum checksum)
throws IOException {
LZ4Factory factory = LZ4Factory.fastestInstance();
LZ4Compressor compressor = factory.fastCompressor();
int blockSize = RaftSharedContext.BLOCK_SIZE;
try (FileOutputStream fos = new FileOutputStream(outputFile);
CheckedOutputStream cos = new CheckedOutputStream(fos, checksum);
BufferedOutputStream bos = new BufferedOutputStream(cos);
LZ4BlockOutputStream lz4os = new LZ4BlockOutputStream(bos,
blockSize,
compressor);
TarArchiveOutputStream tos = new TarArchiveOutputStream(lz4os)) {
Path source = Paths.get(rootDir, sourceDir);
CompressUtil.tarDir(source, tos);
tos.flush();
fos.getFD().sync();
}
}
private static void tarDir(Path source, TarArchiveOutputStream tos)
throws IOException {
Files.walkFileTree(source, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult preVisitDirectory(Path dir,
BasicFileAttributes attrs)
throws IOException {
String entryName = buildTarEntryName(source, dir);
if (!entryName.isEmpty()) {
TarArchiveEntry entry = new TarArchiveEntry(dir.toFile(),
entryName);
tos.putArchiveEntry(entry);
tos.closeArchiveEntry();
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFile(Path file,
BasicFileAttributes attributes)
throws IOException {
// Only copy files, no symbolic links
if (attributes.isSymbolicLink()) {
return FileVisitResult.CONTINUE;
}
String targetFile = buildTarEntryName(source, file);
TarArchiveEntry entry = new TarArchiveEntry(file.toFile(),
targetFile);
tos.putArchiveEntry(entry);
Files.copy(file, tos);
tos.closeArchiveEntry();
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException e) {
return FileVisitResult.TERMINATE;
}
});
}
private static String buildTarEntryName(Path topLevel, Path current) {
return topLevel.getFileName().resolve(topLevel.relativize(current))
.toString();
}
public static void decompressTar(String sourceFile, String outputDir,
Checksum checksum) throws IOException {
Path source = Paths.get(sourceFile);
Path target = Paths.get(outputDir);
if (Files.notExists(source)) {
throw new IOException(String.format(
"The source file %s doesn't exists", source));
}
LZ4Factory factory = LZ4Factory.fastestInstance();
LZ4FastDecompressor decompressor = factory.fastDecompressor();
try (InputStream fis = Files.newInputStream(source);
CheckedInputStream cis = new CheckedInputStream(fis, checksum);
BufferedInputStream bis = new BufferedInputStream(cis);
LZ4BlockInputStream lz4is = new LZ4BlockInputStream(bis,
decompressor);
TarArchiveInputStream tis = new TarArchiveInputStream(lz4is)) {
ArchiveEntry entry;
while ((entry = tis.getNextEntry()) != null) {
// Create a new path, zip slip validate
Path newPath = zipSlipProtect(entry, target);
if (entry.isDirectory()) {
Files.createDirectories(newPath);
} else {
// check parent folder again
Path parent = newPath.getParent();
if (parent != null) {
if (Files.notExists(parent)) {
Files.createDirectories(parent);
}
}
// Copy TarArchiveInputStream to Path newPath
Files.copy(tis, newPath,
StandardCopyOption.REPLACE_EXISTING);
}
}
}
}
private static Path zipSlipProtect(ArchiveEntry entry, Path targetDir)
throws IOException {
Path targetDirResolved = targetDir.resolve(entry.getName());
/*
* Make sure normalized file still has targetDir as its prefix,
* else throws exception
*/
Path normalizePath = targetDirResolved.normalize();
if (!normalizePath.startsWith(targetDir)) {
throw new IOException(String.format("Bad entry: %s",
entry.getName()));
}
return normalizePath;
}
public static void compressZip(String rootDir, String sourceDir,
String outputFile, Checksum checksum)
throws IOException {
try (FileOutputStream fos = new FileOutputStream(outputFile);
CheckedOutputStream cos = new CheckedOutputStream(fos, checksum);
BufferedOutputStream bos = new BufferedOutputStream(cos);
ZipOutputStream zos = new ZipOutputStream(bos)) {
CompressUtil.zipDir(rootDir, sourceDir, zos);
zos.flush();
fos.getFD().sync();
}
}
private static void zipDir(String rootDir, String sourceDir,
ZipOutputStream zos) throws IOException {
String dir = Paths.get(rootDir, sourceDir).toString();
File[] files = new File(dir).listFiles();
E.checkNotNull(files, "files");
for (File file : files) {
String child = Paths.get(sourceDir, file.getName()).toString();
if (file.isDirectory()) {
zipDir(rootDir, child, zos);
} else {
zos.putNextEntry(new ZipEntry(child));
try (FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis)) {
IOUtils.copy(bis, zos);
}
}
}
}
public static void decompressZip(String sourceFile, String outputDir,
Checksum checksum) throws IOException {
try (FileInputStream fis = new FileInputStream(sourceFile);
CheckedInputStream cis = new CheckedInputStream(fis, checksum);
BufferedInputStream bis = new BufferedInputStream(cis);
ZipInputStream zis = new ZipInputStream(bis)) {
ZipEntry entry;
while ((entry = zis.getNextEntry()) != null) {
String fileName = entry.getName();
File entryFile = new File(Paths.get(outputDir, fileName)
.toString());
FileUtils.forceMkdir(entryFile.getParentFile());
try (FileOutputStream fos = new FileOutputStream(entryFile);
BufferedOutputStream bos = new BufferedOutputStream(fos)) {
IOUtils.copy(zis, bos);
bos.flush();
fos.getFD().sync();
}
}
/*
* Continue to read all remaining bytes(extra metadata of ZipEntry)
* directly from the checked stream, Otherwise, the checksum value
* maybe unexpected.
* See https://coderanch.com/t/279175/java/ZipInputStream
*/
IOUtils.copy(cis, NullOutputStream.NULL_OUTPUT_STREAM);
}
}
}

View File

@ -1,101 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.baidu.hugegraph.util;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.zip.CheckedInputStream;
import java.util.zip.CheckedOutputStream;
import java.util.zip.Checksum;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.io.output.NullOutputStream;
public final class ZipUtil {
public static void compress(String rootDir, String sourceDir,
String outputFile, Checksum checksum)
throws IOException {
try (FileOutputStream fos = new FileOutputStream(outputFile);
CheckedOutputStream cos = new CheckedOutputStream(fos, checksum);
BufferedOutputStream bos = new BufferedOutputStream(cos);
ZipOutputStream zos = new ZipOutputStream(bos)) {
ZipUtil.compressDirectoryToZipFile(rootDir, sourceDir, zos);
zos.flush();
fos.getFD().sync();
}
}
private static void compressDirectoryToZipFile(String rootDir,
String sourceDir,
ZipOutputStream zos)
throws IOException {
String dir = Paths.get(rootDir, sourceDir).toString();
File[] files = new File(dir).listFiles();
E.checkNotNull(files, "files");
for (File file : files) {
String child = Paths.get(sourceDir, file.getName()).toString();
if (file.isDirectory()) {
compressDirectoryToZipFile(rootDir, child, zos);
} else {
zos.putNextEntry(new ZipEntry(child));
try (FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis)) {
IOUtils.copy(bis, zos);
}
}
}
}
public static void decompress(String sourceFile, String outputDir,
Checksum checksum) throws IOException {
try (FileInputStream fis = new FileInputStream(sourceFile);
CheckedInputStream cis = new CheckedInputStream(fis, checksum);
BufferedInputStream bis = new BufferedInputStream(cis);
ZipInputStream zis = new ZipInputStream(bis)) {
ZipEntry entry;
while ((entry = zis.getNextEntry()) != null) {
String fileName = entry.getName();
File entryFile = new File(Paths.get(outputDir, fileName)
.toString());
FileUtils.forceMkdir(entryFile.getParentFile());
try (FileOutputStream fos = new FileOutputStream(entryFile);
BufferedOutputStream bos = new BufferedOutputStream(fos)) {
IOUtils.copy(zis, bos);
bos.flush();
fos.getFD().sync();
}
}
// Continue to read all remaining bytes(extra metadata of ZipEntry)
// directly from the checked stream, Otherwise, the checksum value
// maybe unexpected.
//
// See https://coderanch.com/t/279175/java/ZipInputStream
IOUtils.copy(cis, NullOutputStream.NULL_OUTPUT_STREAM);
}
}
}

View File

@ -154,8 +154,10 @@ function free_memory() {
free=`expr $mem_free + $mem_buffer + $mem_cached`
free=`expr $free / 1024`
elif [ "$os" == "Darwin" ]; then
free=`top -l 1 | head -n 10 | grep PhysMem | awk -F',' '{print $2}' \
| awk -F'M' '{print $1}' | tr -d " "`
local pages_free=`vm_stat | awk '/Pages free/{print $0}' | awk -F'[:.]+' '{print $2}' | tr -d " "`
local pages_inactive=`vm_stat | awk '/Pages inactive/{print $0}' | awk -F'[:.]+' '{print $2}' | tr -d " "`
local pages_available=`expr $pages_free + $pages_inactive`
free=`expr $pages_available \* 4096 / 1024 / 1024`
else
echo "Unsupported operating system $os"
exit 1

View File

@ -1,7 +1,7 @@
{
"subject": "hugegraph-license",
"public_alias": "publiccert",
"store_password": "803b6cc3-d144-47e8-948f-ec8b39c8881e",
"store_ticket": "803b6cc3-d144-47e8-948f-ec8b39c8881e",
"publickey_path": "/public-certs.store",
"license_path": "conf/hugegraph-community.license"
}

View File

@ -79,9 +79,11 @@ public abstract class RocksDBSessions extends BackendSessionPool {
public abstract void merge(String table, byte[] key, byte[] value);
public abstract void increase(String table, byte[] key, byte[] value);
public abstract void remove(String table, byte[] key);
public abstract void delete(String table, byte[] keyFrom, byte[] keyTo);
public abstract void delete(String table, byte[] key);
public abstract void deleteSingle(String table, byte[] key);
public abstract void deletePrefix(String table, byte[] key);
public abstract void deleteRange(String table,
byte[] keyFrom, byte[] keyTo);
public abstract byte[] get(String table, byte[] key);

View File

@ -512,7 +512,7 @@ public class RocksDBStdSessions extends RocksDBSessions {
E.checkArgument(compressions.isEmpty() ||
compressions.size() == numLevels,
"Elements number of '%s' must be 0 or " +
"be the same as '%s', bug got %s != %s",
"be the same as '%s', but got %s != %s",
RocksDBOptions.LEVELS_COMPRESSIONS.name(),
RocksDBOptions.NUM_LEVELS.name(),
compressions.size(), numLevels);
@ -840,7 +840,20 @@ public class RocksDBStdSessions extends RocksDBSessions {
* Delete a record by key from a table
*/
@Override
public void remove(String table, byte[] key) {
public void delete(String table, byte[] key) {
try (CFHandle cf = cf(table)) {
this.batch.delete(cf.get(), key);
} catch (RocksDBException e) {
throw new BackendException(e);
}
}
/**
* Delete the only one version of a record by key from a table
* NOTE: requires that the key exists and was not overwritten.
*/
@Override
public void deleteSingle(String table, byte[] key) {
try (CFHandle cf = cf(table)) {
this.batch.singleDelete(cf.get(), key);
} catch (RocksDBException e) {
@ -852,7 +865,7 @@ public class RocksDBStdSessions extends RocksDBSessions {
* Delete a record by key(or prefix with key) from a table
*/
@Override
public void delete(String table, byte[] key) {
public void deletePrefix(String table, byte[] key) {
byte[] keyFrom = key;
byte[] keyTo = Arrays.copyOf(key, key.length);
keyTo = BinarySerializer.increaseOne(keyTo);
@ -867,7 +880,7 @@ public class RocksDBStdSessions extends RocksDBSessions {
* Delete a range of keys from a table
*/
@Override
public void delete(String table, byte[] keyFrom, byte[] keyTo) {
public void deleteRange(String table, byte[] keyFrom, byte[] keyTo) {
try (CFHandle cf = cf(table)) {
this.batch.deleteRange(cf.get(), keyFrom, keyTo);
} catch (RocksDBException e) {

View File

@ -103,7 +103,7 @@ public class RocksDBTable extends BackendTable<Session, BackendEntry> {
} else {
for (BackendColumn col : entry.columns()) {
assert entry.belongToMe(col) : entry;
session.remove(this.table(), col.name);
session.delete(this.table(), col.name);
}
}
}
@ -271,7 +271,7 @@ public class RocksDBTable extends BackendTable<Session, BackendEntry> {
}
protected static final long sizeOfBackendEntry(BackendEntry entry) {
return BinaryEntryIterator.sizeOfBackendEntry(entry);
return BinaryEntryIterator.sizeOfEntry(entry);
}
private static class RocksDBShardSpliter extends ShardSpliter<Session> {
@ -327,7 +327,7 @@ public class RocksDBTable extends BackendTable<Session, BackendEntry> {
@Override
public byte[] position(String position) {
if (END.equals(position)) {
if (START.equals(position) || END.equals(position)) {
return null;
}
return StringEncoding.decodeBase64(position);

View File

@ -75,7 +75,20 @@ public class RocksDBTables {
}
}
public static class VertexLabel extends RocksDBTable {
public static class SchemaTable extends RocksDBTable {
public SchemaTable(String database, String table) {
super(database, table);
}
@Override
public void delete(Session session, BackendEntry entry) {
assert entry.columns().isEmpty();
session.deletePrefix(this.table(), entry.id().asBytes());
}
}
public static class VertexLabel extends SchemaTable {
public static final String TABLE = HugeType.VERTEX_LABEL.string();
@ -84,7 +97,7 @@ public class RocksDBTables {
}
}
public static class EdgeLabel extends RocksDBTable {
public static class EdgeLabel extends SchemaTable {
public static final String TABLE = HugeType.EDGE_LABEL.string();
@ -93,7 +106,7 @@ public class RocksDBTables {
}
}
public static class PropertyKey extends RocksDBTable {
public static class PropertyKey extends SchemaTable {
public static final String TABLE = HugeType.PROPERTY_KEY.string();
@ -102,7 +115,7 @@ public class RocksDBTables {
}
}
public static class IndexLabel extends RocksDBTable {
public static class IndexLabel extends SchemaTable {
public static final String TABLE = HugeType.INDEX_LABEL.string();
@ -168,7 +181,7 @@ public class RocksDBTables {
*/
for (BackendEntry.BackendColumn column : entry.columns()) {
// Don't assert entry.belongToMe(column), length-prefix is 1*
session.delete(this.table(), column.name);
session.deletePrefix(this.table(), column.name);
}
}
}

View File

@ -325,24 +325,33 @@ public class RocksDBSstSessions extends RocksDBSessions {
* Delete a record by key from a table
*/
@Override
public void remove(String table, byte[] key) {
throw new NotSupportException("RocksDBSstStore remove()");
public void delete(String table, byte[] key) {
throw new NotSupportException("RocksDBSstStore delete()");
}
/**
* Delete the only one version of a record by key from a table
* NOTE: requires that the key exists and was not overwritten.
*/
@Override
public void deleteSingle(String table, byte[] key) {
throw new NotSupportException("RocksDBSstStore deleteSingle()");
}
/**
* Delete a record by key(or prefix with key) from a table
*/
@Override
public void delete(String table, byte[] key) {
throw new NotSupportException("RocksDBSstStore delete()");
public void deletePrefix(String table, byte[] key) {
throw new NotSupportException("RocksDBSstStore deletePrefix()");
}
/**
* Delete a range of keys from a table
*/
@Override
public void delete(String table, byte[] keyFrom, byte[] keyTo) {
throw new NotSupportException("RocksDBSstStore delete()");
public void deleteRange(String table, byte[] keyFrom, byte[] keyTo) {
throw new NotSupportException("RocksDBSstStore deleteRange()");
}
/**

View File

@ -107,7 +107,7 @@ public class EdgeApiTest extends BaseApiTest {
// Now allowed to modify sortkey values, the property 'date' has changed
content = assertResponseStatus(400, r);
Assert.assertTrue(content.contains(
"either be null or equal to origin when " +
"either be null or equal to the origin value when " +
"specified edge id"));
// Update edge without edgeId

View File

@ -5114,9 +5114,15 @@ public class EdgeCoreTest extends BaseCoreTest {
while (page != null) {
GraphTraversal<?, ?> iterator = graph.traversal().V(james).bothE()
.has("~page", page).limit(1);
Assert.assertEquals(1, IteratorUtils.count(iterator));
long size = IteratorUtils.count(iterator);
if (size == 0L) {
// The last page is empty
Assert.assertEquals(6, count);
} else {
Assert.assertEquals(1, size);
}
page = TraversalUtil.page(iterator);
count++;
count += size;
}
Assert.assertEquals(6, count);
}
@ -5134,9 +5140,15 @@ public class EdgeCoreTest extends BaseCoreTest {
while (page != null) {
GraphTraversal<?, ?> iterator = graph.traversal().V(james).outE()
.has("~page", page).limit(1);
Assert.assertEquals(1, IteratorUtils.count(iterator));
long size = IteratorUtils.count(iterator);
if (size == 0L) {
// The last page is empty
Assert.assertEquals(4, count);
} else {
Assert.assertEquals(1, size);
}
page = TraversalUtil.page(iterator);
count++;
count += size;
}
Assert.assertEquals(4, count);
}
@ -5154,9 +5166,15 @@ public class EdgeCoreTest extends BaseCoreTest {
while (page != null) {
GraphTraversal<?, ?> iterator = graph.traversal().V(james).inE()
.has("~page", page).limit(1);
Assert.assertEquals(1, IteratorUtils.count(iterator));
long size = IteratorUtils.count(iterator);
if (size == 0L) {
// The last page is empty
Assert.assertEquals(2, count);
} else {
Assert.assertEquals(1, size);
}
page = TraversalUtil.page(iterator);
count++;
count += size;
}
Assert.assertEquals(2, count);
}
@ -6728,7 +6746,48 @@ public class EdgeCoreTest extends BaseCoreTest {
}
@Test
public void testQueryEdgeByPageResultsMatched() {
public void testQueryEdgeByPageResultsMatchedAll() {
Assume.assumeTrue("Not support paging",
storeFeatures().supportsQueryByPage());
HugeGraph graph = graph();
init100LookEdges();
List<Edge> all = graph.traversal().E().toList();
GraphTraversal<Edge, Edge> iter;
String page = PageInfo.PAGE_NONE;
int size = 21;
Set<Edge> pageAll = new HashSet<>();
for (int i = 0; i < 100 / size; i++) {
iter = graph.traversal().E()
.has("~page", page).limit(size);
@SuppressWarnings("unchecked")
List<Edge> edges = IteratorUtils.asList(iter);
Assert.assertEquals(size, edges.size());
pageAll.addAll(edges);
page = TraversalUtil.page(iter);
}
iter = graph.traversal().E()
.has("~page", page).limit(size);
@SuppressWarnings("unchecked")
List<Edge> edges = IteratorUtils.asList(iter);
Assert.assertEquals(16, edges.size());
pageAll.addAll(edges);
page = TraversalUtil.page(iter);
Assert.assertEquals(100, pageAll.size());
Assert.assertTrue(all.containsAll(pageAll));
Assert.assertNull(page);
}
@Test
public void testQueryEdgeByPageResultsMatchedAllWithFullPage() {
Assume.assumeTrue("Not support paging",
storeFeatures().supportsQueryByPage());
@ -6756,6 +6815,15 @@ public class EdgeCoreTest extends BaseCoreTest {
}
Assert.assertEquals(100, pageAll.size());
Assert.assertTrue(all.containsAll(pageAll));
if (page != null) {
iter = graph.traversal().E().has("~page", page);
long count = IteratorUtils.count(iter);
Assert.assertEquals(0L, count);
page = TraversalUtil.page(iter);
CloseableIterator.closeIterator(iter);
}
Assert.assertNull(page);
}
@ -6962,9 +7030,10 @@ public class EdgeCoreTest extends BaseCoreTest {
int count = 0;
while (page != null) {
GraphTraversal<?, ?> iterator = fetcher.apply(page);
Assert.assertEquals(1, IteratorUtils.count(iterator));
long size = IteratorUtils.count(iterator);
Assert.assertLte(1L, size);
page = TraversalUtil.page(iterator);
count++;
count += size;
}
return count;
}

View File

@ -765,6 +765,31 @@ public class EdgeLabelCoreTest extends SchemaCoreTest {
}
@Test
public void testAppendEdgeLabelWithSkAsNullableProperties() {
super.initPropertyKeys();
SchemaManager schema = graph().schema();
schema.vertexLabel("person")
.properties("name", "age", "city")
.primaryKeys("name")
.create();
schema.vertexLabel("book").properties("id", "name")
.primaryKeys("id").create();
schema.edgeLabel("look")
.properties("time")
.link("person", "book")
.multiTimes().sortKeys("time")
.create();
Assert.assertThrows(IllegalArgumentException.class, () -> {
schema.edgeLabel("look")
.properties("time", "weight")
.nullableKeys("time", "weight")
.append();
});
}
@Test
public void testRemoveEdgeLabel() {
super.initPropertyKeys();

View File

@ -255,9 +255,9 @@ public class TaskCoreTest extends BaseCoreTest {
+ "schema.propertyKey('lang').asText().ifNotExist().create();"
+ "schema.propertyKey('date').asDate().ifNotExist().create();"
+ "schema.propertyKey('price').asInt().ifNotExist().create();"
+ "person1=schema.vertexLabel('person1').properties('name','age').ifNotExist().create();"
+ "person2=schema.vertexLabel('person2').properties('name','age').ifNotExist().create();"
+ "knows=schema.edgeLabel('knows').sourceLabel('person1').targetLabel('person2').properties('date').ifNotExist().create();"
+ "schema.vertexLabel('person1').properties('name','age').ifNotExist().create();"
+ "schema.vertexLabel('person2').properties('name','age').ifNotExist().create();"
+ "schema.edgeLabel('knows').sourceLabel('person1').targetLabel('person2').properties('date').ifNotExist().create();"
+ "for(int i = 0; i < 1000; i++) {"
+ " p1=graph.addVertex(T.label,'person1','name','p1-'+i,'age',29);"
+ " p2=graph.addVertex(T.label,'person2','name','p2-'+i,'age',27);"
@ -302,6 +302,89 @@ public class TaskCoreTest extends BaseCoreTest {
Assert.assertEquals("[1000]", task.result());
}
@Test
public void testGremlinJobWithSerializedResults() throws TimeoutException {
HugeGraph graph = graph();
TaskScheduler scheduler = graph.taskScheduler();
String script = "schema=graph.schema();"
+ "schema.propertyKey('name').asText().ifNotExist().create();"
+ "schema.vertexLabel('char').useCustomizeNumberId().properties('name').ifNotExist().create();"
+ "schema.edgeLabel('next').sourceLabel('char').targetLabel('char').properties('name').ifNotExist().create();"
+ "g.addV('char').property(id,1).property('name','A').as('a')"
+ " .addV('char').property(id,2).property('name','B').as('b')"
+ " .addV('char').property(id,3).property('name','C').as('c')"
+ " .addV('char').property(id,4).property('name','D').as('d')"
+ " .addV('char').property(id,5).property('name','E').as('e')"
+ " .addV('char').property(id,6).property('name','F').as('f')"
+ " .addE('next').from('a').to('b').property('name','ab')"
+ " .addE('next').from('b').to('c').property('name','bc')"
+ " .addE('next').from('b').to('d').property('name','bd')"
+ " .addE('next').from('c').to('d').property('name','cd')"
+ " .addE('next').from('c').to('e').property('name','ce')"
+ " .addE('next').from('d').to('e').property('name','de')"
+ " .addE('next').from('e').to('f').property('name','ef')"
+ " .addE('next').from('f').to('d').property('name','fd')"
+ " .iterate();"
+ "g.tx().commit(); g.E().count();";
HugeTask<Object> task = runGremlinJob(script);
task = scheduler.waitUntilTaskCompleted(task.id(), 10);
Assert.assertEquals("test-gremlin-job", task.name());
Assert.assertEquals("gremlin", task.type());
Assert.assertEquals(TaskStatus.SUCCESS, task.status());
Assert.assertEquals("[8]", task.result());
Id edgeLabelId = graph.schema().getEdgeLabel("next").id();
script = "g.V(1).outE().inV().path()";
task = runGremlinJob(script);
task = scheduler.waitUntilTaskCompleted(task.id(), 10);
Assert.assertEquals(TaskStatus.SUCCESS, task.status());
String expected = String.format("[{\"labels\":[[],[],[]],\"objects\":["
+ "{\"id\":1,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"A\"}},"
+ "{\"id\":\"L1>%s>>L2\",\"label\":\"next\",\"type\":\"edge\",\"outV\":1,\"outVLabel\":\"char\",\"inV\":2,\"inVLabel\":\"char\",\"properties\":{\"name\":\"ab\"}},"
+ "{\"id\":2,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"B\"}}"
+ "]}]", edgeLabelId);
Assert.assertEquals(expected, task.result());
script = "g.V(1).out().out().path()";
task = runGremlinJob(script);
task = scheduler.waitUntilTaskCompleted(task.id(), 10);
Assert.assertEquals(TaskStatus.SUCCESS, task.status());
expected = "[{\"labels\":[[],[],[]],\"objects\":["
+ "{\"id\":1,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"A\"}},"
+ "{\"id\":2,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"B\"}},"
+ "{\"id\":3,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"C\"}}]},"
+ "{\"labels\":[[],[],[]],\"objects\":["
+ "{\"id\":1,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"A\"}},"
+ "{\"id\":2,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"B\"}},"
+ "{\"id\":4,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"D\"}}]}]";
Assert.assertEquals(expected, task.result());
script = "g.V(1).outE().inV().tree()";
task = runGremlinJob(script);
task = scheduler.waitUntilTaskCompleted(task.id(), 10);
Assert.assertEquals(TaskStatus.SUCCESS, task.status());
expected = String.format("[[{\"key\":{\"id\":1,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"A\"}},"
+ "\"value\":["
+ "{\"key\":{\"id\":\"L1>%s>>L2\",\"label\":\"next\",\"type\":\"edge\",\"outV\":1,\"outVLabel\":\"char\",\"inV\":2,\"inVLabel\":\"char\",\"properties\":{\"name\":\"ab\"}},"
+ "\"value\":[{\"key\":{\"id\":2,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"B\"}},\"value\":[]}]}]}]]",
edgeLabelId);
Assert.assertEquals(expected, task.result());
script = "g.V(1).out().out().tree()";
task = runGremlinJob(script);
task = scheduler.waitUntilTaskCompleted(task.id(), 10);
Assert.assertEquals(TaskStatus.SUCCESS, task.status());
expected = "[[{\"key\":{\"id\":1,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"A\"}},"
+ "\"value\":[{\"key\":{\"id\":2,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"B\"}},"
+ "\"value\":["
+ "{\"key\":{\"id\":3,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"C\"}},\"value\":[]},"
+ "{\"key\":{\"id\":4,\"label\":\"char\",\"type\":\"vertex\",\"properties\":{\"name\":\"D\"}},\"value\":[]}]}]}]]";
Assert.assertEquals(expected, task.result());
}
@Test
public void testGremlinJobWithFailure() throws TimeoutException {
HugeGraph graph = graph();

View File

@ -52,6 +52,7 @@ import com.baidu.hugegraph.backend.id.IdGenerator;
import com.baidu.hugegraph.backend.id.SnowflakeIdGenerator;
import com.baidu.hugegraph.backend.id.SplicingIdGenerator;
import com.baidu.hugegraph.backend.page.PageInfo;
import com.baidu.hugegraph.backend.page.PageState;
import com.baidu.hugegraph.backend.query.Condition;
import com.baidu.hugegraph.backend.query.ConditionQuery;
import com.baidu.hugegraph.backend.query.Query;
@ -6456,7 +6457,49 @@ public class VertexCoreTest extends BaseCoreTest {
}
@Test
public void testQueryByPageResultsMatched() {
public void testQueryByPageResultsMatchedAll() {
Assume.assumeTrue("Not support paging",
storeFeatures().supportsQueryByPage());
HugeGraph graph = graph();
init100Books();
List<Vertex> all = graph.traversal().V().toList();
GraphTraversal<Vertex, Vertex> iter;
String page = PageInfo.PAGE_NONE;
int size = 22;
Set<Vertex> pageAll = new HashSet<>();
for (int i = 0; i < 100 / size; i++) {
iter = graph.traversal().V()
.has("~page", page).limit(size);
@SuppressWarnings("unchecked")
List<Vertex> vertices = IteratorUtils.asList(iter);
Assert.assertEquals(size, vertices.size());
pageAll.addAll(vertices);
page = TraversalUtil.page(iter);
CloseableIterator.closeIterator(iter);
}
iter = graph.traversal().V()
.has("~page", page).limit(size);
@SuppressWarnings("unchecked")
List<Vertex> vertices = IteratorUtils.asList(iter);
Assert.assertEquals(12, vertices.size());
pageAll.addAll(vertices);
page = TraversalUtil.page(iter);
Assert.assertEquals(100, pageAll.size());
Assert.assertTrue(all.containsAll(pageAll));
Assert.assertNull(page);
}
@Test
public void testQueryByPageResultsMatchedAllWithFullPage() {
Assume.assumeTrue("Not support paging",
storeFeatures().supportsQueryByPage());
@ -6485,6 +6528,15 @@ public class VertexCoreTest extends BaseCoreTest {
}
Assert.assertEquals(100, pageAll.size());
Assert.assertTrue(all.containsAll(pageAll));
if (page != null) {
iter = graph.traversal().V().has("~page", page);
long count = IteratorUtils.count(iter);
Assert.assertEquals(0L, count);
page = TraversalUtil.page(iter);
CloseableIterator.closeIterator(iter);
}
Assert.assertNull(page);
}
@ -6511,6 +6563,35 @@ public class VertexCoreTest extends BaseCoreTest {
});
}
@Test
public void testQueryByPageWithSpecialBase64Chars() {
Assume.assumeTrue("Not support paging",
storeFeatures().supportsQueryByPage());
final String pageWith3Base64Chars = "AAAAADsyABwAEAqI546LS6WW57unBgA" +
"EAAAAAPB////+8H////4alhxAZS8va6" +
"opcAKpklipAAQAAAAAAAAAAQ==";
final String pageWithSpace = "AAAAADsyABwAEAqI546LS6WW57unBgAEAAAAAP" +
"B//// 8H////4alhxAZS8va6opcAKpklipAAQA" +
"AAAAAAAAAQ==";
HugeGraph graph = graph();
init100Books();
// Contains valid character '+' and '/' and '='
GraphTraversal<Vertex, Vertex> traversal;
traversal = graph.traversal().V()
.has("~page", pageWith3Base64Chars).limit(10);
Assert.assertNotNull(traversal);
CloseableIterator.closeIterator(traversal);
// Contains invalid base64 character ' ', will be replaced to '+'
traversal = graph.traversal().V()
.has("~page", pageWithSpace).limit(10);
Assert.assertNotNull(traversal);
CloseableIterator.closeIterator(traversal);
}
@Test
public void testQueryByPageWithInvalidLimit() {
Assume.assumeTrue("Not support paging",

View File

@ -737,6 +737,23 @@ public class VertexLabelCoreTest extends SchemaCoreTest {
});
}
@Test
public void testAppendVertexLabelWithPkAsNullableProperties() {
super.initPropertyKeys();
SchemaManager schema = graph().schema();
schema.vertexLabel("person")
.properties("name", "age")
.primaryKeys("name")
.create();
Assert.assertThrows(IllegalArgumentException.class, () -> {
schema.vertexLabel("person")
.properties("name", "city")
.nullableKeys("name", "city")
.append();
});
}
@Test
public void testRemoveVertexLabel() {
super.initPropertyKeys();

View File

@ -44,6 +44,7 @@ import com.baidu.hugegraph.perf.PerfUtil.Watched;
import com.baidu.hugegraph.schema.PropertyKey;
import com.baidu.hugegraph.schema.SchemaManager;
import com.baidu.hugegraph.task.TaskScheduler;
import com.baidu.hugegraph.testutil.Whitebox;
import com.baidu.hugegraph.type.define.IdStrategy;
import com.baidu.hugegraph.type.define.NodeRole;
import com.google.common.collect.ImmutableSet;
@ -251,7 +252,8 @@ public class TestGraph implements Graph {
@SuppressWarnings({ "unchecked", "rawtypes" })
@Override
public <I extends Io> I io(final Io.Builder<I> builder) {
HugeGraphSONModule.OPTIMIZE_SERIALIZE = false;
Whitebox.setInternalState(HugeGraphSONModule.class,
"OPTIMIZE_SERIALIZE", false);
return (I) builder.graph(this).onMapper(mapper ->
mapper.addRegistry(HugeGraphIoRegistry.instance())
).create();

View File

@ -37,6 +37,7 @@ import com.baidu.hugegraph.unit.core.DataTypeTest;
import com.baidu.hugegraph.unit.core.DirectionsTest;
import com.baidu.hugegraph.unit.core.ExceptionTest;
import com.baidu.hugegraph.unit.core.LocksTableTest;
import com.baidu.hugegraph.unit.core.PageStateTest;
import com.baidu.hugegraph.unit.core.QueryTest;
import com.baidu.hugegraph.unit.core.RangeTest;
import com.baidu.hugegraph.unit.core.RolePermissionTest;
@ -59,6 +60,7 @@ import com.baidu.hugegraph.unit.serializer.SerializerFactoryTest;
import com.baidu.hugegraph.unit.serializer.StoreSerializerTest;
import com.baidu.hugegraph.unit.serializer.TableBackendEntryTest;
import com.baidu.hugegraph.unit.serializer.TextBackendEntryTest;
import com.baidu.hugegraph.unit.util.CompressUtilTest;
import com.baidu.hugegraph.unit.util.JsonUtilTest;
import com.baidu.hugegraph.unit.util.StringEncodingTest;
import com.baidu.hugegraph.unit.util.VersionTest;
@ -98,6 +100,7 @@ import com.baidu.hugegraph.unit.util.VersionTest;
ExceptionTest.class,
BackendStoreSystemInfoTest.class,
TraversalUtilTest.class,
PageStateTest.class,
/* serializer */
BytesBufferTest.class,
@ -123,7 +126,8 @@ import com.baidu.hugegraph.unit.util.VersionTest;
/* utils */
VersionTest.class,
JsonUtilTest.class,
StringEncodingTest.class
StringEncodingTest.class,
CompressUtilTest.class
})
public class UnitTestSuite {
}

View File

@ -0,0 +1,90 @@
/*
* Copyright 2017 HugeGraph Authors
*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with this
* work for additional information regarding copyright ownership. The ASF
* licenses this file to You under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations
* under the License.
*/
package com.baidu.hugegraph.unit.core;
import java.util.Arrays;
import org.junit.Test;
import com.baidu.hugegraph.backend.BackendException;
import com.baidu.hugegraph.backend.page.PageState;
import com.baidu.hugegraph.testutil.Assert;
public class PageStateTest {
private String pageWith3Base64Chars = "AAAAADsyABwAEAqI546LS6WW57unBgA" +
"EAAAAAPB////+8H////4alhxAZS8va6" +
"opcAKpklipAAQAAAAAAAAAAQ==";
private String pageWithSpace = "AAAAADsyABwAEAqI546LS6WW57unBgAEAAAAAP" +
"B//// 8H////4alhxAZS8va6opcAKpklipAAQA" +
"AAAAAAAAAQ==";
@Test
public void testOriginalStringPageToBytes() {
byte[] validPage = PageState.toBytes(pageWith3Base64Chars);
Assert.assertNotNull(validPage);
Assert.assertThrows(BackendException.class, () -> {
PageState.toBytes(pageWithSpace);
}, e -> {
Assert.assertContains("Invalid page:", e.toString());
});
}
@Test
public void testDecodePageWithSpecialBase64Chars() {
// Assert decode '+' and '/' and '=' and space successfully
Assert.assertNotNull(PageState.fromString(pageWith3Base64Chars));
byte[] decodePlus = PageState.fromString(pageWith3Base64Chars)
.position();
byte[] decodeSpace = PageState.fromString(pageWithSpace).position();
Assert.assertTrue(Arrays.equals(decodePlus, decodeSpace));
}
@Test
public void testDecodePageWithInvalidStringPage() {
final String invalidPageWithBase64Chars = "dGVzdCBiYXNlNjQ=";
Assert.assertThrows(BackendException.class, () -> {
PageState.fromString(invalidPageWithBase64Chars);
}, e -> {
Assert.assertContains("Invalid page: '0x", e.toString());
});
final String invalidBase64Chars = "!abc~";
Assert.assertThrows(BackendException.class, () -> {
PageState.fromString(invalidBase64Chars);
}, e -> {
Assert.assertContains("Invalid page:", e.toString());
});
}
@Test
public void testEmptyPageState() {
Assert.assertEquals(0, PageState.EMPTY.offset());
Assert.assertNull(PageState.EMPTY.toString());
Assert.assertEquals(PageState.EMPTY,
PageState.fromBytes(PageState.EMPTY_BYTES));
}
}

View File

@ -77,7 +77,8 @@ public class BaseRocksDBUnitTest extends BaseUnitTest {
protected void clearData() throws RocksDBException {
for (String table : new ArrayList<>(this.rocks.openedTables())) {
this.rocks.session().delete(table, new byte[]{0}, new byte[]{-1});
this.rocks.session().deleteRange(table,
new byte[]{0}, new byte[]{-1});
}
this.commit();
}

View File

@ -141,7 +141,7 @@ public class RocksDBPerfTest extends BaseRocksDBUnitTest {
for (int i = 0; i < n; i++) {
int value = comms.get(i);
String old = String.format("index:%3d:%d", i, value);
session.remove(TABLE, b(old));
session.delete(TABLE, b(old));
value = r.nextInt(n); // TODO: aggregate
value = i + 1;

View File

@ -285,7 +285,7 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
Assert.assertEquals("19", get("person:1gage"));
Assert.assertEquals("Beijing", get("person:1gcity"));
this.rocks.session().remove(TABLE, b("person:1gage"));
this.rocks.session().delete(TABLE, b("person:1gage"));
this.commit();
Assert.assertEquals("James", get("person:1gname"));
@ -303,7 +303,7 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
Assert.assertEquals("19", get("person:1gage"));
Assert.assertEquals("Beijing", get("person:1gcity"));
this.rocks.session().remove(TABLE, b("person:1"));
this.rocks.session().delete(TABLE, b("person:1"));
this.commit();
Assert.assertEquals("James", get("person:1gname"));
@ -325,7 +325,7 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
Assert.assertEquals("19", get("person:1gage"));
Assert.assertEquals("Beijing", get("person:1gcity"));
this.rocks.session().delete(TABLE, b("person:1"));
this.rocks.session().deletePrefix(TABLE, b("person:1"));
this.commit();
Assert.assertEquals(null, get("person:1gname"));
@ -353,7 +353,7 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
Assert.assertEquals("Lisa", get("person:2gname"));
Assert.assertEquals("Hebe", get("person:3gname"));
this.rocks.session().delete(TABLE, b("person:1"), b("person:3"));
this.rocks.session().deleteRange(TABLE, b("person:1"), b("person:3"));
this.commit();
Assert.assertEquals(null, get("person:1gname"));
@ -385,7 +385,7 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
byte[] value21 = b("value-2-1");
session.put(TABLE, key21, value21);
session.delete(TABLE, key11, new byte[]{1, 3});
session.deleteRange(TABLE, key11, new byte[]{1, 3});
this.commit();
Assert.assertArrayEquals(null, session.get(TABLE, key11));
@ -409,14 +409,14 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
byte[] value21 = b("value-2-1");
session.put(TABLE, key21, value21);
session.delete(TABLE, new byte[]{1, -3}, new byte[]{1, 3});
session.deleteRange(TABLE, new byte[]{1, -3}, new byte[]{1, 3});
this.commit();
Assert.assertArrayEquals(value11, session.get(TABLE, key11));
Assert.assertArrayEquals(value12, session.get(TABLE, key12));
Assert.assertArrayEquals(value21, session.get(TABLE, key21));
session.delete(TABLE, new byte[]{1, 1}, new byte[]{1, -1});
session.deleteRange(TABLE, new byte[]{1, 1}, new byte[]{1, -1});
this.commit();
Assert.assertArrayEquals(null, session.get(TABLE, key11));
@ -448,7 +448,8 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
byte[] value20 = b("value-2-0");
session.put(TABLE, key20, value20);
session.delete(TABLE, new byte[]{1, 0}, new byte[]{1, (byte) 0xff});
session.deleteRange(TABLE,
new byte[]{1, 0}, new byte[]{1, (byte) 0xff});
this.commit();
Assert.assertArrayEquals(null, session.get(TABLE, key11));
@ -457,7 +458,8 @@ public class RocksDBSessionsTest extends BaseRocksDBUnitTest {
Assert.assertArrayEquals(value14, session.get(TABLE, key14));
Assert.assertArrayEquals(value20, session.get(TABLE, key20));
session.delete(TABLE, new byte[]{1, (byte) 0xff}, new byte[]{2, 0});
session.deleteRange(TABLE,
new byte[]{1, (byte) 0xff}, new byte[]{2, 0});
this.commit();
Assert.assertArrayEquals(null, session.get(TABLE, key11));

View File

@ -0,0 +1,134 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.baidu.hugegraph.unit.util;
import java.io.File;
import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.Arrays;
import java.util.zip.Checksum;
import org.apache.commons.io.FileUtils;
import org.junit.Test;
import com.alipay.sofa.jraft.util.CRC64;
import com.baidu.hugegraph.util.CompressUtil;
import com.google.common.collect.ImmutableList;
import junit.framework.AssertionFailedError;
public class CompressUtilTest {
@Test
public void testZipCompress() throws IOException {
String rootDir = "temp";
// temp/ss
String sourceDir = "ss";
String zipFile = "output.zip";
String output = "output";
try {
prepareFiles(rootDir, sourceDir);
Checksum checksum = new CRC64();
CompressUtil.compressZip(rootDir, sourceDir, zipFile, checksum);
CompressUtil.decompressZip(zipFile, output, checksum);
assertDirEquals(rootDir, output);
} finally {
FileUtils.deleteQuietly(new File(rootDir));
FileUtils.deleteQuietly(new File(zipFile));
FileUtils.deleteQuietly(new File(output));
}
}
@Test
public void testTarCompress() throws IOException {
String rootDir = "temp";
// temp/ss
String sourceDir = "ss";
String tarFile = "output.tar";
String output = "output";
try {
prepareFiles(rootDir, sourceDir);
Checksum checksum = new CRC64();
CompressUtil.compressTar(rootDir, sourceDir, tarFile, checksum);
CompressUtil.decompressTar(tarFile, output, checksum);
assertDirEquals(rootDir, output);
} finally {
FileUtils.deleteQuietly(new File(rootDir));
FileUtils.deleteQuietly(new File(tarFile));
FileUtils.deleteQuietly(new File(output));
}
}
private static void prepareFiles(String rootDir, String sourceDir)
throws IOException {
// temp/ss/g
String gDir = Paths.get(rootDir, sourceDir, "g").toString();
File file1 = new File(gDir, "file-1");
FileUtils.writeLines(file1, ImmutableList.of("g1-aaa", "g1-bbb"));
File file2 = new File(gDir, "file-2");
FileUtils.writeLines(file2, ImmutableList.of("g2-aaa", "g2-bbb"));
// temp/ss/m
String mDir = Paths.get(rootDir, sourceDir, "m").toString();
file1 = new File(mDir, "file-1");
FileUtils.writeLines(file1, ImmutableList.of("m1-aaa", "m1-bbb"));
file2 = new File(mDir, "file-2");
FileUtils.writeLines(file2, ImmutableList.of("m2-aaa", "m2-bbb"));
// temp/ss/s
String sDir = Paths.get(rootDir, sourceDir, "s").toString();
file1 = new File(sDir, "file-1");
FileUtils.writeLines(file1, ImmutableList.of("s1-aaa", "s1-bbb"));
file2 = new File(sDir, "file-2");
FileUtils.writeLines(file2, ImmutableList.of("s2-aaa", "s2-bbb"));
}
private static void assertDirEquals(String expect, String actual)
throws IOException {
Path expectDir = Paths.get(expect);
Path actualDir = Paths.get(actual);
Files.walkFileTree(expectDir, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path file,
BasicFileAttributes attrs)
throws IOException {
FileVisitResult result = super.visitFile(file, attrs);
// Get the relative file name from path "one"
Path relativize = expectDir.relativize(file);
// Construct the path for the counterpart file in "other"
Path fileInOther = actualDir.resolve(relativize);
byte[] theseBytes = Files.readAllBytes(file);
byte[] otherBytes = Files.readAllBytes(fileInOther);
if (!Arrays.equals(theseBytes, otherBytes)) {
throw new AssertionFailedError(file + " is not equal to " +
fileInOther);
}
return result;
}
});
}
}

View File

@ -96,7 +96,7 @@
<compiler.target>1.8</compiler.target>
<slf4j.version>1.7.5</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<log4j2.version>2.12.1</log4j2.version>
<log4j2.version>2.17.0</log4j2.version>
<junit.version>4.12</junit.version>
<tinkerpop.version>3.4.3</tinkerpop.version>
<commons.io.version>2.4</commons.io.version>