feat: added the OpenTelemetry trace support (#2477)

TODO: we need enhance our shell experience --------- Co-authored-by: imbajin <jin@apache.org> Co-authored-by: VGalaxies <vgalaxies@apache.org>
2024-03-15 17:30:16 +08:00 · 2024-03-15 17:30:16 +08:00 · 7b33574fd0
parent c09982b0df
commit 7b33574fd0
11 changed files with 306 additions and 37 deletions
--- a/.gitignore
+++ b/.gitignore
@ -83,3 +83,6 @@ hs_err_pid*
 .mtj.tmp/
 # blueJ files
 *.ctxt
+
+# docker volumes ignore
+hugegraph-server/hugegraph-dist/docker/data/
--- a/README.md
+++ b/README.md
@ -16,7 +16,9 @@
 ## What is Apache HugeGraph?

 [HugeGraph](https://hugegraph.apache.org/) is a fast-speed and highly-scalable [graph database](https://en.wikipedia.org/wiki/Graph_database). 
-Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability. As compliance to [Apache TinkerPop 3](https://tinkerpop.apache.org/) framework, various complicated graph queries can be accomplished through [Gremlin](https://tinkerpop.apache.org/gremlin.html)(a powerful graph traversal language).
+Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability. 
+As compliance to [Apache TinkerPop 3](https://tinkerpop.apache.org/) framework, various complicated graph queries can be 
+achieved through [Gremlin](https://tinkerpop.apache.org/gremlin.html)(a powerful graph traversal language).


 ## Features
@ -40,7 +42,7 @@ You can visit [doc page](https://hugegraph.apache.org/docs/quickstart/hugegraph-
 >  
 > 1. The docker image of hugegraph is a convenience release, but not **official distribution** artifacts. You can find more details from [ASF Release Distribution Policy](https://infra.apache.org/release-distribution.html#dockerhub).
 > 
-> 2. Recommand to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
+> 2. Recommend to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.

 ### 2. Download Way

--- a/hugegraph-server/hugegraph-dist/docker/README.md
+++ b/hugegraph-server/hugegraph-dist/docker/README.md
@ -4,11 +4,11 @@
 > 
 > 1. The docker image of hugegraph is a convenience release, not official distribution artifacts from ASF. You can find more details from [ASF Release Distribution Policy](https://infra.apache.org/release-distribution.html#dockerhub).
 > 
-> 2. Recommand to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
+> 2. Recommend to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.

 ## 1. Deploy

-We can use docker to quickly start an inner HugeGraph server with RocksDB in background.
+We can use docker to quickly start an inner HugeGraph server with RocksDB in the background.

 1. Using docker run

@ -29,9 +29,9 @@ We can use docker to quickly start an inner HugeGraph server with RocksDB in bac

 ## 2. Create Sample Graph on Server Startup

-If you want to **pre-load** some (test) data or graphs in container(by default), you can set the env `PRELOAD=ture`
+If you want to **preload** some (test) data or graphs in container(by default), you can set the env `PRELOAD=ture`

-If you want to customize the pre-loaded data, please mount the the groovy scripts (not necessary).
+If you want to customize the preloaded data, please mount the groovy scripts (not necessary).

 1. Using docker run

@ -67,7 +67,7 @@ If you want to customize the pre-loaded data, please mount the the groovy script

 2. Using docker compose

-    Similarly, we can set the envionment variables in the docker-compose.yaml:
+    Similarly, we can set the environment variables in the docker-compose.yaml:

    ```yaml
    version: '3'
@ -81,3 +81,34 @@ If you want to customize the pre-loaded data, please mount the the groovy script
          - AUTH=true
          - PASSWORD=123456
    ```
+## 4. Running Open-Telemetry-Collector
+
+> CAUTION:
+> 
+> The `docker-compose-trace.yaml` utilizes `Grafana` and `Grafana-Tempo`, both of them are licensed under [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.en.html), you should be aware of and use them with caution. Currently, we mainly provide this template for everyone to **test**
+>
+1. Start Open-Telemetry-Collector
+
+    ```bash
+    cd hugegraph-server/hugegraph-dist/docker/example
+    docker-compose -f docker-compose-trace.yaml -p hugegraph-trace up -d
+    ```
+   
+2. Active Open-Telemetry-Agent
+
+    ```bash
+    ./start-hugegraph.sh -y true
+    ```
+   
+3. Stop Open-Telemetry-Collector
+
+    ```bash
+    cd hugegraph-server/hugegraph-dist/docker/example
+    docker-compose -f docker-compose-trace.yaml -p hugegraph-trace stop
+    ```
+   
+4. References
+
+   - [What is OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/)
+
+   - [Tempo in Grafana](https://grafana.com/docs/tempo/latest/getting-started/tempo-in-grafana/)
--- a/hugegraph-server/hugegraph-dist/docker/conf/grafana-datasources.yaml
+++ b/hugegraph-server/hugegraph-dist/docker/conf/grafana-datasources.yaml
@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: 1
+
+datasources:
+- name: Prometheus
+  type: prometheus
+  uid: prometheus
+  access: proxy
+  orgId: 1
+  url: http://prometheus:9090
+  basicAuth: false
+  isDefault: false
+  version: 1
+  editable: false
+  jsonData:
+    httpMethod: GET
+- name: Tempo
+  type: tempo
+  access: proxy
+  orgId: 1
+  url: http://tempo:3200
+  basicAuth: false
+  isDefault: true
+  version: 1
+  editable: false
+  apiVersion: 1
+  uid: tempo
+  jsonData:
+    httpMethod: GET
+    serviceMap:
+      datasourceUid: prometheus
--- a/hugegraph-server/hugegraph-dist/docker/conf/prometheus.yaml
+++ b/hugegraph-server/hugegraph-dist/docker/conf/prometheus.yaml
@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+global:
+  scrape_interval:     15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+  - job_name: 'tempo'
+    static_configs:
+      - targets: [ 'tempo:3200' ]
--- a/hugegraph-server/hugegraph-dist/docker/conf/tempo.yaml
+++ b/hugegraph-server/hugegraph-dist/docker/conf/tempo.yaml
@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+stream_over_http_enabled: true
+server:
+  http_listen_port: 3200
+  log_level: info
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        http:
+        grpc:
+
+compactor:
+  compaction:
+    block_retention: 48h # configure total trace retention here
+
+metrics_generator:
+  registry:
+    external_labels:
+      source: tempo
+
+  storage:
+    path: /tmp/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+
+storage:
+  trace:
+    backend: local                     # backend configuration to use
+    wal:
+      path: /tmp/tempo/wal             # where to store the wal locally
+    local:
+      path: /tmp/tempo/blocks
+
+overrides:
+  defaults:
+    metrics_generator:
+      processors: [ service-graphs, span-metrics ] # enables metrics generator
--- a/hugegraph-server/hugegraph-dist/docker/example/docker-compose-trace.yaml
+++ b/hugegraph-server/hugegraph-dist/docker/example/docker-compose-trace.yaml
@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  tempo:
+    image: grafana/tempo:latest
+    command: ["-config.file=/etc/tempo.yaml"]
+    volumes:
+      - ../conf/tempo.yaml:/etc/tempo.yaml
+      - ../data/tempo:/tmp/tempo
+    ports:
+      - "3200:3200"  # tempo
+      - "9095:9095"  # tempo grpc
+      - "4317:4317"  # otlp grpc
+      - "4318:4318"  # otlp http
+
+  prometheus:
+    image: prom/prometheus:latest
+    command:
+      - --config.file=/etc/prometheus.yaml
+      - --web.enable-remote-write-receiver
+      - --enable-feature=exemplar-storage
+      - --storage.tsdb.path=/tmp/data
+    volumes:
+      - ../conf/prometheus.yaml:/etc/prometheus.yaml
+      - ../data/prometheus:/tmp/data
+    ports:
+      - "9090:9090"
+
+  grafana:
+    image: grafana/grafana:10.2.2
+    volumes:
+      - ../conf/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor traceQLStreaming metricsSummary
+    ports:
+      - "3000:3000"
--- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh
@ -15,6 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
 function abs_path() {
    SOURCE="${BASH_SOURCE[0]}"
    while [[ -h "$SOURCE" ]]; do
@ -39,24 +40,20 @@ EXT="$TOP/ext"
 PLUGINS="$TOP/plugins"
 LOGS="$TOP/logs"
 OUTPUT=${LOGS}/hugegraph-server.log
+GITHUB="https://github.com"

 export HUGEGRAPH_HOME="$TOP"
 . "${BIN}"/util.sh

-GREMLIN_SERVER_CONF="$1"
-REST_SERVER_CONF="$2"
-OPEN_SECURITY_CHECK="$3"
-
-if [[ $# -eq 3 ]]; then
-    USER_OPTION=""
-    GC_OPTION=""
-elif [[ $# -eq 4 ]]; then
-    USER_OPTION="$4"
-    GC_OPTION=""
-elif [[ $# -eq 5 ]]; then
-    USER_OPTION="$4"
-    GC_OPTION="$5"
-fi
+# Parse the server arguments in array way
+SERVER_ARGS=("$@")
+GREMLIN_SERVER_CONF="${SERVER_ARGS[0]:-}"
+REST_SERVER_CONF="${SERVER_ARGS[1]:-}"
+OPEN_SECURITY_CHECK="${SERVER_ARGS[2]:-}"
+# Param will be empty str("") if not set
+USER_OPTION="${SERVER_ARGS[3]:-}"
+GC_OPTION="${SERVER_ARGS[4]:-}"
+OPEN_TELEMETRY="${SERVER_ARGS[5]:-}"

 ensure_path_writable "$LOGS"
 ensure_path_writable "$PLUGINS"
@ -64,11 +61,12 @@ ensure_path_writable "$PLUGINS"
 # The maximum and minimum heap memory that service can use
 MAX_MEM=$((32 * 1024))
 MIN_MEM=$((1 * 512))
+# TODO: upgrade to Java 11 in 1.5.0
 MIN_JAVA_VERSION=8

-# download binary file
+# Note: Download for HTTPS, could comment out if you don't need it
 if [[ ! -e "${CONF}/hugegraph-server.keystore" ]]; then
-    download "${CONF}" "https://github.com/apache/hugegraph-doc/raw/binary-1.0/dist/server/hugegraph-server.keystore"
+    download "${CONF}" "${GITHUB}/apache/hugegraph-doc/raw/binary-1.0/dist/server/hugegraph-server.keystore"
 fi

 # Add the slf4j-log4j12 binding
@ -81,7 +79,7 @@ CP="$CP":$(find -L $LIB -name '*.jar' \
    \! -name 'log4j-slf4j-impl*.jar' | sort | tr '\n' ':')
 # Add the jars in ext (at any subdirectory depth)
 CP="$CP":$(find -L $EXT -name '*.jar' | sort | tr '\n' ':')
-# Add the jars in plugins (at any subdirectory depth)
+# Add the jars in plugins (at any subdirectory depth), check "javaagent" related jars carefully
 CP="$CP":$(find -L $PLUGINS -name '*.jar' | sort | tr '\n' ':')

 # (Cygwin only) Use ; classpath separator and reformat paths for Windows ("C:\foo")
@ -153,6 +151,44 @@ if [[ ${OPEN_SECURITY_CHECK} == "true" ]]; then
    JVM_OPTIONS="${JVM_OPTIONS} -Djava.security.manager=org.apache.hugegraph.security.HugeSecurityManager"
 fi

+if [ "${OPEN_TELEMETRY}" == "true" ]; then
+    OT_JAR="opentelemetry-javaagent.jar"
+    OT_JAR_PATH="${PLUGINS}/${OT_JAR}"
+
+    if [[ ! -e "${OT_JAR_PATH}" ]]; then
+        echo "## Downloading ${OT_JAR}..."
+        download "${PLUGINS}" \
+            "${GITHUB}/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.1.0/${OT_JAR}"
+
+        if [[ ! -e "${OT_JAR_PATH}" ]]; then
+            echo "## Error: Failed to download ${OT_JAR}." >>${OUTPUT}
+            exit 1
+        fi
+    fi
+
+    # Note: remember update it if we change the jar 
+    expected_md5="e3bcbbe8ed9b6d840fa4c333b36f369f"
+    actual_md5=$(md5sum "${OT_JAR_PATH}" | awk '{print $1}')
+
+    if [[ "${expected_md5}" != "${actual_md5}" ]]; then
+        echo "## Error: MD5 checksum verification failed for ${OT_JAR_PATH}." >>${OUTPUT}
+        echo "## Tips: Remove the file and try again." >>${OUTPUT}
+        exit 1
+    fi
+
+    # Note: check carefully if multi "javeagent" params are set
+    export JAVA_TOOL_OPTIONS="-javaagent:${PLUGINS}/${OT_JAR}"
+    export OTEL_TRACES_EXPORTER=otlp
+    export OTEL_METRICS_EXPORTER=none
+    export OTEL_LOGS_EXPORTER=none
+    export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
+    # 127.0.0.1:4317 is the port of otel-collector running in Docker located in
+    # 'hugegraph-server/hugegraph-dist/docker/example/docker-compose-trace.yaml'.
+    # Make sure the otel-collector is running before starting HugeGraphServer.
+    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4317
+    export OTEL_RESOURCE_ATTRIBUTES=service.name=server
+fi
+
 # Turn on security check
 exec ${JAVA} -Dname="HugeGraphServer" ${JVM_OPTIONS} ${JAVA_OPTIONS} -cp ${CLASSPATH}: \
    org.apache.hugegraph.dist.HugeGraphServer ${GREMLIN_SERVER_CONF} ${REST_SERVER_CONF} \
--- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh
@ -15,14 +15,18 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
 OPEN_MONITOR="false"
 OPEN_SECURITY_CHECK="true"
+# change to "true" to enable telemetry(Trace) by default
+OPEN_TELEMETRY="false"
 DAEMON="true"
 #VERBOSE=""
 GC_OPTION=""
 USER_OPTION=""
 SERVER_STARTUP_TIMEOUT_S=30
-# todo: move abs_path funtion to shell like util.sh
+
+# TODO: move abs_path function to shell like util.sh
 function abs_path() {
    SOURCE="${BASH_SOURCE[0]}"
    while [[ -h "$SOURCE" ]]; do
@ -42,7 +46,8 @@ PID_FILE="$BIN/pid"

 . "$BIN"/util.sh

-while getopts "d:g:m:p:s:j:t:v" arg; do
+# Note: keep ':' in the end of the string to indicate the option needs a value
+while getopts "d:g:m:p:s:j:t:y:" arg; do
    case ${arg} in
        d) DAEMON="$OPTARG" ;;
        g) GC_OPTION="$OPTARG" ;;
@ -51,8 +56,8 @@ while getopts "d:g:m:p:s:j:t:v" arg; do
        s) OPEN_SECURITY_CHECK="$OPTARG" ;;
        j) USER_OPTION="$OPTARG" ;;
        t) SERVER_STARTUP_TIMEOUT_S="$OPTARG" ;;
-        # TODO: should remove it in future (check the usage carefully)
-        v) VERBOSE="verbose" ;;
+        # Telemetry is used to collect metrics, traces and logs
+        y) OPEN_TELEMETRY="$OPTARG" ;;
        # Note: update usage info when the params changed
        ?) exit_with_usage_help ;;
    esac
@ -82,21 +87,24 @@ fi
 GREMLIN_SERVER_CONF="gremlin-server.yaml"
 if [[ $PRELOAD == "true" ]]; then
    GREMLIN_SERVER_CONF="gremlin-server-preload.yaml"
-    EXAMPLE_SCRPIT="example-preload.groovy"
+    EXAMPLE_SCRIPT="example-preload.groovy"
    cp "${CONF}"/gremlin-server.yaml "${CONF}/${GREMLIN_SERVER_CONF}"
-    cp "${SCRIPTS}"/example.groovy "${SCRIPTS}/${EXAMPLE_SCRPIT}"
-    sed -i -e "s/empty-sample.groovy/$EXAMPLE_SCRPIT/g" "${CONF}/${GREMLIN_SERVER_CONF}"
-    sed -i -e '/registerRocksDB/d; /serverStarted/d' "${SCRIPTS}/${EXAMPLE_SCRPIT}"
+    cp "${SCRIPTS}"/example.groovy "${SCRIPTS}/${EXAMPLE_SCRIPT}"
+    sed -i -e "s/empty-sample.groovy/$EXAMPLE_SCRIPT/g" "${CONF}/${GREMLIN_SERVER_CONF}"
+    sed -i -e '/registerRocksDB/d; /serverStarted/d' "${SCRIPTS}/${EXAMPLE_SCRIPT}"
 fi

+# TODO: show the output message in hugegraph-server.sh when start the server
 if [[ $DAEMON == "true" ]]; then
    echo "Starting HugeGraphServer in daemon mode..."
    "${BIN}"/hugegraph-server.sh "${CONF}/${GREMLIN_SERVER_CONF}" "${CONF}"/rest-server.properties \
-    "${OPEN_SECURITY_CHECK}" "${USER_OPTION}" "${GC_OPTION}" >>"${LOGS}"/hugegraph-server.log 2>&1 &
+    "${OPEN_SECURITY_CHECK}" "${USER_OPTION}" "${GC_OPTION}" "${OPEN_TELEMETRY}" \
+    >>"${LOGS}"/hugegraph-server.log 2>&1 &
 else
    echo "Starting HugeGraphServer in foreground mode..."
    "${BIN}"/hugegraph-server.sh "${CONF}/${GREMLIN_SERVER_CONF}" "${CONF}"/rest-server.properties \
-    "${OPEN_SECURITY_CHECK}" "${USER_OPTION}" "${GC_OPTION}" >>"${LOGS}"/hugegraph-server.log 2>&1
+    "${OPEN_SECURITY_CHECK}" "${USER_OPTION}" "${GC_OPTION}" "${OPEN_TELEMETRY}" \
+    >>"${LOGS}"/hugegraph-server.log 2>&1
 fi

 PID="$!"
--- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh
@ -15,6 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
 function command_available() {
    local cmd=$1
    if [[ -x "$(command -v "$cmd")" ]]; then
@ -385,6 +386,6 @@ function kill_process_and_wait() {
 }

 function exit_with_usage_help(){
-    echo "USAGE: $0 [-d true|false] [-g g1] [-m true|false] [-p true|false] [-s true|false] [-j java_options] [-t timeout]"
+    echo "USAGE: $0 [-d true|false] [-g g1] [-m true|false] [-p true|false] [-s true|false] [-j java_options] [-t timeout] [-y true|false]"
    exit 1
 }
--- a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-server.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-server.sh
@ -47,7 +47,7 @@ SERIALIZER=${backend_serializer_map[$BACKEND]}
 sed -i "s/backend=.*/backend=$BACKEND/" $CONF
 sed -i "s/serializer=.*/serializer=$SERIALIZER/" $CONF

-# Set PostgreSQL configurations if needed
+# Set PostgresSQL configurations if needed
 if [ "$BACKEND" == "postgresql" ]; then
    sed -i '/org.postgresql.Driver/,+2 s/\#//g' $CONF
 fi
@ -70,4 +70,5 @@ if [ -n "$JACOCO_PORT" ]; then
    JACOCO_OPTION="-javaagent:${JACOCO_JAR}=includes=*,port=${JACOCO_PORT},destfile=jacoco-it.exec,output=tcpserver"
 fi

-echo -e "pa" | $BIN/init-store.sh && $BIN/start-hugegraph.sh -j "$JACOCO_OPTION" -t 60 -v
+echo -e "pa" | $BIN/init-store.sh
+$BIN/start-hugegraph.sh -j "$JACOCO_OPTION" -t 60