Compare commits
1 Commits
| Author | SHA1 | Date |
|---|---|---|
HaiyangDING
|
4221e02342 |
44
README.md
44
README.md
|
|
@ -4,16 +4,12 @@
|
|||
|
||||
The *Alibaba Cluster Trace Program* is published by Alibaba Group. By providing cluster trace from real production, the program helps the researchers, students and people who are interested in the field to get better understanding of the characterastics of modern internet data centers (IDC's) and the workloads.
|
||||
|
||||
So far, four versions of traces have been released:
|
||||
So far, two versions of traces have been released:
|
||||
|
||||
* *cluster-trace-v2017* includes about 1300 machines in a period of 12 hours. The trace-v2017 firstly introduces the collocation of online services (aka long running applications) and batch workloads. To see more about this trace, see related documents ([trace_2017](./cluster-trace-v2017/trace_201708.md)). Download link is available after a short survey ([survey link](https://goo.gl/forms/eOoe6DwZQpd2H5n53)).
|
||||
* *cluster-trace-v2018* includes about 4000 machines in a period of 8 days. Besides having larger scaler than trace-v2017, this piece trace also contains the DAG information of our production batch workloads. See related documents for more details ([trace_2018](./cluster-trace-v2018/trace_2018.md)). Download link is available after a survey (less than a minute, [survey link](http://alibabadeveloper.mikecrm.com/BdJtacN)).
|
||||
* *cluster-trace-gpu-v2020* includes over 6500 GPUs (on ~1800 machines) in a period of 2 months. It describes the AI/ML workloads in the MLaaS (Machine-Learning-as-a-Service) provided by the [Alibaba PAI (Platform for Artificial Intelligence)](https://www.alibabacloud.com/product/machine-learning) on GPU clusters. See the subdirectory ([pai_gpu_trace_2020](./cluster-trace-gpu-v2020/README.md)) for the released data, schema, and scripts for processing and visualization. Our analysis paper published in USENIX NSDI '22 is available [here](https://www.usenix.org/conference/nsdi22/presentation/weng).
|
||||
* *cluster-trace-microservices-v2021* contains 20000+ microservices in a period of 12 hours. The traces the first released to introduce the runtime metrics of microservices in the production cluster, including call dependencies, respond time, call rates, and so on. See the subdirectory ([trace_2021](./cluster-trace-microservices-v2021/README.md)) for more details. Our analysis paper, accepted by SoCC '21, is available [here](http://cloud.siat.ac.cn/pdca/socc2021-AlibabaTraceAnalysis.pdf).
|
||||
* *cluster-trace-microarchitecture-v2022* first provides AMTrace (Alibaba Microarchitecutre Trace). AMTrace is the first fine-granulairty and large-scale microarchitectural metrics of Alibaba Colocation Datacenter. Based AMTrace, researchers can analysis: CPU performance, microarchitecture contention, memory bandwidth contention and so on. [Our paper](https://doi.org/10.1145/3545008.3545026) is accepted by ICPP'22. See the subdirectory ([trace_2022](./cluster-trace-microarchitecture-v2022/README.md)) for more details.
|
||||
* *cluster-trace-gpu-v2023* includes over 6200 GPUs (on ~1200 machines). It describes the AI/ML workloads with diverse resource specifications in a heterogeneous GPU cluster. In our "[Beware of Fragmentation](https://www.usenix.org/conference/atc23/presentation/weng)" paper (published in USENIX ATC '23), we modeled this trace in a [Kubernetes Scheduler Simulator](https://github.com/hkust-adsl/kubernetes-scheduler-simulator) and demonstrated that our proposed Fragmentation Gradient Descent (FGD) policy outperforms classic scheduling policies like Best-Fit, Dot-Product, etc. See [fgd_gpu_trace_2023](./cluster-trace-gpu-v2023/README.md) for the released data, schema, and scripts for processing.
|
||||
* *cluster-trace-v2018* includes about 4000 machines in a perids of 8 days. Besides having larger scaler than trace-v2017, this piece trace also contains the DAG information of our production batch workloads. See related documents for more details ([trace_2018](./cluster-trace-v2018/trace_2018.md)). Download link is available after a survey (less than a minute, [survey link](http://alibabadeveloper.mikecrm.com/BdJtacN)).
|
||||
|
||||
We encourage anyone to use the traces for study or research purposes, and if you had any question when using the trace, please contact us via email: [alibaba-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com), or file an issue on Github. Filing an issue is recommanded as the discussion would help all the community. Note that the more clearly you ask the question, the more likely you would get a clear answer.
|
||||
We encourage anyone to use the traces for study or research purposes, and if you had any question when using the trace, please contact us via email: [aliababa-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com), or file an issue on Github. Filing an issue is recommanded as the discussion would help all the community. Note that the more clearly you ask the question, the more likely you would get a clear answer.
|
||||
|
||||
It would be much appreciated if you could tell us once any publication using our trace is available, as we are maintaining a list of related publicatioins for more researchers to better communicate with each other.
|
||||
|
||||
|
|
@ -27,7 +23,7 @@ From our perspective, the data is provided to address [the challenges Alibaba fa
|
|||
|
||||
1. **Workload characterizations**. How to characterize Alibaba workloads in a way that we can simulate various production workload in a representative way for scheduling and resource management strategy studies.
|
||||
2. **New algorithms to assign workload to machines**. How to assign and reschedule workloads to machines for better resource utilization and ensuring the performance SLA for different applications (e.g. by reducing resource contention and defining proper proirities).
|
||||
3. **Collaboration between online service scheduler (Sigma) and batch jobs scheduler (Fuxi)**. How to adjust resource allocation between online service and batch jobs to improve throughput of batch jobs while maintain acceptable QoS (Quality of Service) and fast failure recovery for online service. As the scale of collocation (workloads managed by different schedulers) keeps growing, the design of collaboration mechanism is becoming more and more critical.
|
||||
3. **Collaboration between online service scheduler (Sigma) and batch jobs scheduler (Fuxi)**. How to adjust resource allocation between online service and batch jobs to improve throughput of batch jobs while maintain acceptable QoS (Quolity of Service) and fast failure recovery for online service. As the scale of collocation (workloads managed by different schedulers) keeps growing, the design of collaboration mechanism is becoming more and more critical.
|
||||
|
||||
Last but not least, we are always open to work together with researchers to improve the efficiency of our clusters, and there are positions open for research interns. If you had any idea in your mind, please contact us via [aliababa-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com) or [Haiyang Ding](mailto:haiyang.dhy@alibaba-inc.com) (Haiyang maintains this cluster trace and works for Alibaba's resource management & scheduling group).
|
||||
|
||||
|
|
@ -35,34 +31,16 @@ Last but not least, we are always open to work together with researchers to impr
|
|||
|
||||
### Papers using Alibaba cluster trace
|
||||
|
||||
The fundamental idea of our releasing cluster data is to enable researchers & practitioners doing resaerch, simulation with more realistic data and thus making the result closer to industry adoption. It is a huge encouragement to us to see more works using our data. Here is a list of existing works using Alibaba cluster data. **If your paper uses our trace, it would be great if you let us know by sending us email** ([aliababa-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com)).
|
||||
The fundemental idea of our releasing cluster data is to enable researchers & practitioners doing resaerch, simulation with more realistic data and thus making the result closer to industry adoption. It is a huge encouragement to us to see more works using our data. Here is a list of existing works using Alibaba cluster data. If your paper uses our trace, it would be great if you let us know by sending us email ([aliababa-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com)).
|
||||
|
||||
|
||||
* cluster trace GPU v2023
|
||||
* [Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent](https://www.usenix.org/conference/atc23/presentation/weng), Qizhen Weng* and Lingyun Yang* (co-first author), Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. In the 2023 USENIX Annual Technical Conference (ATC '23), Boston, MA, USA, July 2023.
|
||||
* microarchitecture trace v2022
|
||||
* [Characterizing Job Microarchitectural Profiles at Alibaba Scale: Dataset and Analysis](https://doi.org/10.1145/3545008.3545026), Kangjin Wang, Ying Li, Cheng Wang, Tong Jia, Kingsum Chow, Yang Wen, Yaoyong Dou, Guoyao Xu, Chuanjia Hou, Jie Yao, and Liping Zhang. In 51st International Conference on Parallel Processing (ICPP ’22), August 29-September 1, 2022, Bordeaux, France. ACM, New York, NY, USA, 11 pages.
|
||||
* microservices trace v2021
|
||||
* [Characterizing Microservice Dependency and Performance: Alibaba Trace Analysis](http://cloud.siat.ac.cn/pdca/socc2021-AlibabaTraceAnalysis.pdf), Shutian Luo, Huanle Xu, Chengzhi Lu, Kejiang Ye, Guoyao Xu, Liping Zhang, Yu Ding, Jian He, Chengzhong Xu. SoCC'21
|
||||
* [μBench: an open-source factory of benchmark microservice applications](http://netgroup.uniroma2.it/Andrea_Detti/papers/journals/Bench_an_open-source_factory_of_benchmark_microservice_applications.pdf), A. Detti, L. Funari and L. Petrucci, IEEE Transactions on Parallel and Distributed Systems
|
||||
* cluster trace GPU v2020
|
||||
* [MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters](https://www.usenix.org/conference/nsdi22/presentation/weng), Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. In the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22), Renton, WA, USA, April 2022.
|
||||
* cluster trace v2018
|
||||
* [Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces](https://dl.acm.org/citation.cfm?doid=3326285.3329074), Jing Guo, Zihao Chang, Sa Wang, Haiyang Ding, Yihui Feng, Liang Mao, Yungang Bao, IEEE/ACM International Symposium on Quality of Service, IWQoS 2019
|
||||
* [DeepJS: Job Scheduling Based on Deep Reinforcement Learning in Cloud Data Center](https://github.com/RobertLexis/CloudSimPy/blob/master/playground/paper/F0049-4.19.pdf), by Fengcun Li and Bo Hu.
|
||||
* There is an interesting simulator released with this paper: CloudSimPy. You can check it at [CloudSimPy](https://github.com/RobertLexis/CloudSimPy)
|
||||
* Characterizing and Synthesizing Task Dependencies of Data-Parallel Jobs in Alibaba Cloud, by Huangshi Tian, Yunchuan Zheng, and Wei Wang, to appear in ACM Symposium on Cloud Computing (SoCC '19), Santa Cruz, California, November 2019.
|
||||
* [Aladdin: Optimized Maximum Flow Management for Shared Production Clusters](https://ieeexplore.ieee.org/abstract/document/8821038), Heng WU, Wenbo ZHANG, Yuanjia XU, Hao XIANG, Tao HUANG, Haiyang DING, Zheng ZHANG, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS).
|
||||
* cluster trace v2017
|
||||
* [LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation](https://www.usenix.org/system/files/osdi18-shan.pdf), Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang, Purdue University. OSDI'18 (Best paper award!)
|
||||
* [The Elasticity and Plasticity in Semi-Containerized Co-locating Cloud Workload: a View from Alibaba Trace](https://dl.acm.org/citation.cfm?id=3267830), Qixiao Liu and Zhibin Yu. SoCC2018
|
||||
* Zeno: A Straggler Diagnosis System for Distributed Computing Using Machine Learning, Huanxing Shen and Cong Li, Proceedings of the Thirty-Third International Conference, ISC High Performance 2018
|
||||
* [Characterizing Co-located Datacenter Workloads: An Alibaba Case Study](https://arxiv.org/pdf/1808.02919.pdf), Yue Cheng, Zheng Chai, Ali Anwar. APSys2018
|
||||
* [Imbalance in the Cloud: an Analysis on Alibaba Cluster Trace, Chengzhi Lu et al. BIGDATA 2017](http://cloud.siat.ac.cn/~ye/Imbalance_Ye_2017.pdf)
|
||||
* Jiang C, Han G, Lin J, et al. [Characteristics of Co-allocated Online Services and Batch Jobs in Internet Data Centers: A Case Study from Alibaba Cloud[J]](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8636497). IEEE Access, 2019, 7: 22495-22508.
|
||||
* LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation, Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang, Purdue University. OSDI'18 (Best paper award!)
|
||||
* The Elasticity and Plasticity in Semi-Containerized Co-locating Cloud Workload: a View from Alibaba Trace, Qixiao Liu and Zhibin Yu. SoCC2018
|
||||
* Zeno: A Straggler Diagnosis System for Distributed Computing Using Machine Learning, Huanxing Shen and Cong Li, Proceedings of the Thirty-Third International Conference, ISC High Performance 2018
|
||||
* Characterizing Co-located Datacenter Workloads: An Alibaba Case Study, Yue Cheng, Zheng Chai, Ali Anwar. APSys2018
|
||||
* [Imbalance in the Cloud: an Analysis on Alibaba Cluster Trace, Chengzhi Lu et al. BIGDATA 2017](http://cloud.siat.ac.cn/~ye/Imbalance_Ye_2017.pdf)
|
||||
|
||||
### Tech reports and projects on analysing the trace
|
||||
|
||||
So far this session is empty. In future, we are going to link some reports and open source repo on how to anaylsis the trace here, with the permission of the owner.
|
||||
|
||||
The purpose of this is to help more beginners to get start on learning either basic data analysis or how to inspect cluster from statistics perspective.
|
||||
The purpose of this is to help more beginners to get start on learning either basic data analysis or how to inspect cluster from statistics perspective.
|
||||
|
|
@ -1,395 +0,0 @@
|
|||
Attribution 4.0 International
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Corporation ("Creative Commons") is not a law firm and
|
||||
does not provide legal services or legal advice. Distribution of
|
||||
Creative Commons public licenses does not create a lawyer-client or
|
||||
other relationship. Creative Commons makes its licenses and related
|
||||
information available on an "as-is" basis. Creative Commons gives no
|
||||
warranties regarding its licenses, any material licensed under their
|
||||
terms and conditions, or any related information. Creative Commons
|
||||
disclaims all liability for damages resulting from their use to the
|
||||
fullest extent possible.
|
||||
|
||||
Using Creative Commons Public Licenses
|
||||
|
||||
Creative Commons public licenses provide a standard set of terms and
|
||||
conditions that creators and other rights holders may use to share
|
||||
original works of authorship and other material subject to copyright
|
||||
and certain other rights specified in the public license below. The
|
||||
following considerations are for informational purposes only, are not
|
||||
exhaustive, and do not form part of our licenses.
|
||||
|
||||
Considerations for licensors: Our public licenses are
|
||||
intended for use by those authorized to give the public
|
||||
permission to use material in ways otherwise restricted by
|
||||
copyright and certain other rights. Our licenses are
|
||||
irrevocable. Licensors should read and understand the terms
|
||||
and conditions of the license they choose before applying it.
|
||||
Licensors should also secure all rights necessary before
|
||||
applying our licenses so that the public can reuse the
|
||||
material as expected. Licensors should clearly mark any
|
||||
material not subject to the license. This includes other CC-
|
||||
licensed material, or material used under an exception or
|
||||
limitation to copyright. More considerations for licensors:
|
||||
wiki.creativecommons.org/Considerations_for_licensors
|
||||
|
||||
Considerations for the public: By using one of our public
|
||||
licenses, a licensor grants the public permission to use the
|
||||
licensed material under specified terms and conditions. If
|
||||
the licensor's permission is not necessary for any reason--for
|
||||
example, because of any applicable exception or limitation to
|
||||
copyright--then that use is not regulated by the license. Our
|
||||
licenses grant only permissions under copyright and certain
|
||||
other rights that a licensor has authority to grant. Use of
|
||||
the licensed material may still be restricted for other
|
||||
reasons, including because others have copyright or other
|
||||
rights in the material. A licensor may make special requests,
|
||||
such as asking that all changes be marked or described.
|
||||
Although not required by our licenses, you are encouraged to
|
||||
respect those requests where reasonable. More_considerations
|
||||
for the public:
|
||||
wiki.creativecommons.org/Considerations_for_licensees
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Attribution 4.0 International Public License
|
||||
|
||||
By exercising the Licensed Rights (defined below), You accept and agree
|
||||
to be bound by the terms and conditions of this Creative Commons
|
||||
Attribution 4.0 International Public License ("Public License"). To the
|
||||
extent this Public License may be interpreted as a contract, You are
|
||||
granted the Licensed Rights in consideration of Your acceptance of
|
||||
these terms and conditions, and the Licensor grants You such rights in
|
||||
consideration of benefits the Licensor receives from making the
|
||||
Licensed Material available under these terms and conditions.
|
||||
|
||||
|
||||
Section 1 -- Definitions.
|
||||
|
||||
a. Adapted Material means material subject to Copyright and Similar
|
||||
Rights that is derived from or based upon the Licensed Material
|
||||
and in which the Licensed Material is translated, altered,
|
||||
arranged, transformed, or otherwise modified in a manner requiring
|
||||
permission under the Copyright and Similar Rights held by the
|
||||
Licensor. For purposes of this Public License, where the Licensed
|
||||
Material is a musical work, performance, or sound recording,
|
||||
Adapted Material is always produced where the Licensed Material is
|
||||
synched in timed relation with a moving image.
|
||||
|
||||
b. Adapter's License means the license You apply to Your Copyright
|
||||
and Similar Rights in Your contributions to Adapted Material in
|
||||
accordance with the terms and conditions of this Public License.
|
||||
|
||||
c. Copyright and Similar Rights means copyright and/or similar rights
|
||||
closely related to copyright including, without limitation,
|
||||
performance, broadcast, sound recording, and Sui Generis Database
|
||||
Rights, without regard to how the rights are labeled or
|
||||
categorized. For purposes of this Public License, the rights
|
||||
specified in Section 2(b)(1)-(2) are not Copyright and Similar
|
||||
Rights.
|
||||
|
||||
d. Effective Technological Measures means those measures that, in the
|
||||
absence of proper authority, may not be circumvented under laws
|
||||
fulfilling obligations under Article 11 of the WIPO Copyright
|
||||
Treaty adopted on December 20, 1996, and/or similar international
|
||||
agreements.
|
||||
|
||||
e. Exceptions and Limitations means fair use, fair dealing, and/or
|
||||
any other exception or limitation to Copyright and Similar Rights
|
||||
that applies to Your use of the Licensed Material.
|
||||
|
||||
f. Licensed Material means the artistic or literary work, database,
|
||||
or other material to which the Licensor applied this Public
|
||||
License.
|
||||
|
||||
g. Licensed Rights means the rights granted to You subject to the
|
||||
terms and conditions of this Public License, which are limited to
|
||||
all Copyright and Similar Rights that apply to Your use of the
|
||||
Licensed Material and that the Licensor has authority to license.
|
||||
|
||||
h. Licensor means the individual(s) or entity(ies) granting rights
|
||||
under this Public License.
|
||||
|
||||
i. Share means to provide material to the public by any means or
|
||||
process that requires permission under the Licensed Rights, such
|
||||
as reproduction, public display, public performance, distribution,
|
||||
dissemination, communication, or importation, and to make material
|
||||
available to the public including in ways that members of the
|
||||
public may access the material from a place and at a time
|
||||
individually chosen by them.
|
||||
|
||||
j. Sui Generis Database Rights means rights other than copyright
|
||||
resulting from Directive 96/9/EC of the European Parliament and of
|
||||
the Council of 11 March 1996 on the legal protection of databases,
|
||||
as amended and/or succeeded, as well as other essentially
|
||||
equivalent rights anywhere in the world.
|
||||
|
||||
k. You means the individual or entity exercising the Licensed Rights
|
||||
under this Public License. Your has a corresponding meaning.
|
||||
|
||||
|
||||
Section 2 -- Scope.
|
||||
|
||||
a. License grant.
|
||||
|
||||
1. Subject to the terms and conditions of this Public License,
|
||||
the Licensor hereby grants You a worldwide, royalty-free,
|
||||
non-sublicensable, non-exclusive, irrevocable license to
|
||||
exercise the Licensed Rights in the Licensed Material to:
|
||||
|
||||
a. reproduce and Share the Licensed Material, in whole or
|
||||
in part; and
|
||||
|
||||
b. produce, reproduce, and Share Adapted Material.
|
||||
|
||||
2. Exceptions and Limitations. For the avoidance of doubt, where
|
||||
Exceptions and Limitations apply to Your use, this Public
|
||||
License does not apply, and You do not need to comply with
|
||||
its terms and conditions.
|
||||
|
||||
3. Term. The term of this Public License is specified in Section
|
||||
6(a).
|
||||
|
||||
4. Media and formats; technical modifications allowed. The
|
||||
Licensor authorizes You to exercise the Licensed Rights in
|
||||
all media and formats whether now known or hereafter created,
|
||||
and to make technical modifications necessary to do so. The
|
||||
Licensor waives and/or agrees not to assert any right or
|
||||
authority to forbid You from making technical modifications
|
||||
necessary to exercise the Licensed Rights, including
|
||||
technical modifications necessary to circumvent Effective
|
||||
Technological Measures. For purposes of this Public License,
|
||||
simply making modifications authorized by this Section 2(a)
|
||||
(4) never produces Adapted Material.
|
||||
|
||||
5. Downstream recipients.
|
||||
|
||||
a. Offer from the Licensor -- Licensed Material. Every
|
||||
recipient of the Licensed Material automatically
|
||||
receives an offer from the Licensor to exercise the
|
||||
Licensed Rights under the terms and conditions of this
|
||||
Public License.
|
||||
|
||||
b. No downstream restrictions. You may not offer or impose
|
||||
any additional or different terms or conditions on, or
|
||||
apply any Effective Technological Measures to, the
|
||||
Licensed Material if doing so restricts exercise of the
|
||||
Licensed Rights by any recipient of the Licensed
|
||||
Material.
|
||||
|
||||
6. No endorsement. Nothing in this Public License constitutes or
|
||||
may be construed as permission to assert or imply that You
|
||||
are, or that Your use of the Licensed Material is, connected
|
||||
with, or sponsored, endorsed, or granted official status by,
|
||||
the Licensor or others designated to receive attribution as
|
||||
provided in Section 3(a)(1)(A)(i).
|
||||
|
||||
b. Other rights.
|
||||
|
||||
1. Moral rights, such as the right of integrity, are not
|
||||
licensed under this Public License, nor are publicity,
|
||||
privacy, and/or other similar personality rights; however, to
|
||||
the extent possible, the Licensor waives and/or agrees not to
|
||||
assert any such rights held by the Licensor to the limited
|
||||
extent necessary to allow You to exercise the Licensed
|
||||
Rights, but not otherwise.
|
||||
|
||||
2. Patent and trademark rights are not licensed under this
|
||||
Public License.
|
||||
|
||||
3. To the extent possible, the Licensor waives any right to
|
||||
collect royalties from You for the exercise of the Licensed
|
||||
Rights, whether directly or through a collecting society
|
||||
under any voluntary or waivable statutory or compulsory
|
||||
licensing scheme. In all other cases the Licensor expressly
|
||||
reserves any right to collect such royalties.
|
||||
|
||||
|
||||
Section 3 -- License Conditions.
|
||||
|
||||
Your exercise of the Licensed Rights is expressly made subject to the
|
||||
following conditions.
|
||||
|
||||
a. Attribution.
|
||||
|
||||
1. If You Share the Licensed Material (including in modified
|
||||
form), You must:
|
||||
|
||||
a. retain the following if it is supplied by the Licensor
|
||||
with the Licensed Material:
|
||||
|
||||
i. identification of the creator(s) of the Licensed
|
||||
Material and any others designated to receive
|
||||
attribution, in any reasonable manner requested by
|
||||
the Licensor (including by pseudonym if
|
||||
designated);
|
||||
|
||||
ii. a copyright notice;
|
||||
|
||||
iii. a notice that refers to this Public License;
|
||||
|
||||
iv. a notice that refers to the disclaimer of
|
||||
warranties;
|
||||
|
||||
v. a URI or hyperlink to the Licensed Material to the
|
||||
extent reasonably practicable;
|
||||
|
||||
b. indicate if You modified the Licensed Material and
|
||||
retain an indication of any previous modifications; and
|
||||
|
||||
c. indicate the Licensed Material is licensed under this
|
||||
Public License, and include the text of, or the URI or
|
||||
hyperlink to, this Public License.
|
||||
|
||||
2. You may satisfy the conditions in Section 3(a)(1) in any
|
||||
reasonable manner based on the medium, means, and context in
|
||||
which You Share the Licensed Material. For example, it may be
|
||||
reasonable to satisfy the conditions by providing a URI or
|
||||
hyperlink to a resource that includes the required
|
||||
information.
|
||||
|
||||
3. If requested by the Licensor, You must remove any of the
|
||||
information required by Section 3(a)(1)(A) to the extent
|
||||
reasonably practicable.
|
||||
|
||||
4. If You Share Adapted Material You produce, the Adapter's
|
||||
License You apply must not prevent recipients of the Adapted
|
||||
Material from complying with this Public License.
|
||||
|
||||
|
||||
Section 4 -- Sui Generis Database Rights.
|
||||
|
||||
Where the Licensed Rights include Sui Generis Database Rights that
|
||||
apply to Your use of the Licensed Material:
|
||||
|
||||
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
|
||||
to extract, reuse, reproduce, and Share all or a substantial
|
||||
portion of the contents of the database;
|
||||
|
||||
b. if You include all or a substantial portion of the database
|
||||
contents in a database in which You have Sui Generis Database
|
||||
Rights, then the database in which You have Sui Generis Database
|
||||
Rights (but not its individual contents) is Adapted Material; and
|
||||
|
||||
c. You must comply with the conditions in Section 3(a) if You Share
|
||||
all or a substantial portion of the contents of the database.
|
||||
|
||||
For the avoidance of doubt, this Section 4 supplements and does not
|
||||
replace Your obligations under this Public License where the Licensed
|
||||
Rights include other Copyright and Similar Rights.
|
||||
|
||||
|
||||
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
|
||||
|
||||
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
|
||||
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
|
||||
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
|
||||
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
|
||||
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
|
||||
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
|
||||
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
|
||||
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
|
||||
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
|
||||
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
|
||||
|
||||
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
|
||||
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
|
||||
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
|
||||
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
|
||||
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
|
||||
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
|
||||
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
|
||||
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
|
||||
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
|
||||
|
||||
c. The disclaimer of warranties and limitation of liability provided
|
||||
above shall be interpreted in a manner that, to the extent
|
||||
possible, most closely approximates an absolute disclaimer and
|
||||
waiver of all liability.
|
||||
|
||||
|
||||
Section 6 -- Term and Termination.
|
||||
|
||||
a. This Public License applies for the term of the Copyright and
|
||||
Similar Rights licensed here. However, if You fail to comply with
|
||||
this Public License, then Your rights under this Public License
|
||||
terminate automatically.
|
||||
|
||||
b. Where Your right to use the Licensed Material has terminated under
|
||||
Section 6(a), it reinstates:
|
||||
|
||||
1. automatically as of the date the violation is cured, provided
|
||||
it is cured within 30 days of Your discovery of the
|
||||
violation; or
|
||||
|
||||
2. upon express reinstatement by the Licensor.
|
||||
|
||||
For the avoidance of doubt, this Section 6(b) does not affect any
|
||||
right the Licensor may have to seek remedies for Your violations
|
||||
of this Public License.
|
||||
|
||||
c. For the avoidance of doubt, the Licensor may also offer the
|
||||
Licensed Material under separate terms or conditions or stop
|
||||
distributing the Licensed Material at any time; however, doing so
|
||||
will not terminate this Public License.
|
||||
|
||||
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
|
||||
License.
|
||||
|
||||
|
||||
Section 7 -- Other Terms and Conditions.
|
||||
|
||||
a. The Licensor shall not be bound by any additional or different
|
||||
terms or conditions communicated by You unless expressly agreed.
|
||||
|
||||
b. Any arrangements, understandings, or agreements regarding the
|
||||
Licensed Material not stated herein are separate from and
|
||||
independent of the terms and conditions of this Public License.
|
||||
|
||||
|
||||
Section 8 -- Interpretation.
|
||||
|
||||
a. For the avoidance of doubt, this Public License does not, and
|
||||
shall not be interpreted to, reduce, limit, restrict, or impose
|
||||
conditions on any use of the Licensed Material that could lawfully
|
||||
be made without permission under this Public License.
|
||||
|
||||
b. To the extent possible, if any provision of this Public License is
|
||||
deemed unenforceable, it shall be automatically reformed to the
|
||||
minimum extent necessary to make it enforceable. If the provision
|
||||
cannot be reformed, it shall be severed from this Public License
|
||||
without affecting the enforceability of the remaining terms and
|
||||
conditions.
|
||||
|
||||
c. No term or condition of this Public License will be waived and no
|
||||
failure to comply consented to unless expressly agreed to by the
|
||||
Licensor.
|
||||
|
||||
d. Nothing in this Public License constitutes or may be interpreted
|
||||
as a limitation upon, or waiver of, any privileges and immunities
|
||||
that apply to the Licensor or You, including from the legal
|
||||
processes of any jurisdiction or authority.
|
||||
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons is not a party to its public
|
||||
licenses. Notwithstanding, Creative Commons may elect to apply one of
|
||||
its public licenses to material it publishes and in those instances
|
||||
will be considered the “Licensor.” The text of the Creative Commons
|
||||
public licenses is dedicated to the public domain under the CC0 Public
|
||||
Domain Dedication. Except for the limited purpose of indicating that
|
||||
material is shared under a Creative Commons public license or as
|
||||
otherwise permitted by the Creative Commons policies published at
|
||||
creativecommons.org/policies, Creative Commons does not authorize the
|
||||
use of the trademark "Creative Commons" or any other trademark or logo
|
||||
of Creative Commons without its prior written consent including,
|
||||
without limitation, in connection with any unauthorized modifications
|
||||
to any of its public licenses or any other arrangements,
|
||||
understandings, or agreements concerning use of licensed material. For
|
||||
the avoidance of doubt, this paragraph does not form part of the
|
||||
public licenses.
|
||||
|
||||
Creative Commons may be contacted at creativecommons.org.
|
||||
|
|
@ -1,346 +0,0 @@
|
|||
# 1 Introduction
|
||||
|
||||
The released trace contains a hybrid of training and inference jobs running state-of-the-art ML algorithms. It is collected from a large production cluster with over 6,500 GPUs (on ~1800 machines) in [Alibaba PAI (Platform for Artificial Intelligence)](https://www.alibabacloud.com/product/machine-learning), spanning the July and August of 2020.
|
||||
We also include a [Jupyter notebook](./analysis/analysis.ipynb) that parses the trace and highlights some of the main characteristics (see section [3 Demo of Data Analysis](#3-demo-of-data-analysis)).
|
||||
|
||||
|
||||
We also present a characterization study of the trace in a paper, ["MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters"](https://home.cse.ust.hk/~weiwa/papers/weng-nsdi22.pdf), published in [NSDI ’22](https://www.usenix.org/conference/nsdi22). We would encourage anybody who uses this trace to cite our paper.
|
||||
|
||||
```BibTeX
|
||||
@inproceedings{weng2022mlaas,
|
||||
title={{MLaaS} in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous {GPU} Clusters},
|
||||
author={Weng, Qizhen and Xiao, Wencong and Yu, Yinghao and Wang, Wei and Wang, Cheng and He, Jian and Li, Yong and Zhang, Liping and Lin, Wei and Ding, Yu},
|
||||
booktitle={19th $\{$USENIX$\}$ Symposium on Networked Systems Design and Implementation ($\{$NSDI$\}$ 22)},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
- [1 Introduction](#1-introduction)
|
||||
- [2 Data Files in the Trace](#2-data-files-in-the-trace)
|
||||
- [pai_job_table](#pai_job_table)
|
||||
- [pai_task_table](#pai_task_table)
|
||||
- [pai_instance_table](#pai_instance_table)
|
||||
- [pai_sensor_table](#pai_sensor_table)
|
||||
- [pai_group_tag_table](#pai_group_tag_table)
|
||||
- [pai_machine_spec](#pai_machine_spec)
|
||||
- [pai_machine_metric](#pai_machine_metric)
|
||||
- [3 Demo of Data Analysis](#3-demo-of-data-analysis)
|
||||
|
||||
|
||||
# 2 Data Files in the Trace
|
||||
|
||||
|
||||
|
||||
|
||||
The figure above illustrates an architecture overview of PAI, where users submit ML jobs developed in a variety of frameworks (e.g., TensorFlow, PyTorch, Graph-Learn).
|
||||
Upon the job submission, users provide the application code and specify the required compute resources, such as GPUs, CPUs, and memory. Each **job** is translated into multiple **tasks** of different roles, such as parameter servers (PS) and workers for a training job, and evaluator for an inference job. Each task may consist of one or multiple **instances** and can run on multiple machines.
|
||||
PAI employs Docker containers to instantiate tasks for simplified scheduling and execution on heterogeneous hardware.
|
||||
For example, the following job requests 14 CPUs (600% + 400% + 400%) and 1 GPU (0% + 50% + 50%) from the cluster with its three instances.
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
Job_0 --* Task_A
|
||||
Job_0 --* Task_B
|
||||
Task_A --* Instance_A0
|
||||
Task_B --* Instance_B0
|
||||
Task_B --* Instance_B1
|
||||
Job_0 : *job_name* = job_0
|
||||
Job_0 : user
|
||||
Job_0 : ...
|
||||
Task_A : *task_name* = ps
|
||||
Task_A : inst_num = 1
|
||||
Task_A : plan_cpu = 600
|
||||
Task_A : plan_gpu = 0
|
||||
Task_A : ...
|
||||
Instance_A0 : *worker_name* = ps_0
|
||||
Instance_A0 : plan_cpu [inherited] = 600
|
||||
Instance_A0 : plan_gpu [inherited] = 0
|
||||
Instance_A0 : ...
|
||||
Task_B : *task_name* = worker
|
||||
Task_B : inst_num = 2
|
||||
Task_B : plan_cpu = 400
|
||||
Task_B : plan_gpu = 50
|
||||
Task_B : ...
|
||||
Instance_B0 : *worker_name* = worker_0
|
||||
Instance_B0 : plan_cpu [inherited] = 400
|
||||
Instance_B0 : plan_gpu [inherited] = 50
|
||||
Instance_B0 : ...
|
||||
Instance_B1 : *worker_name* = worker_1
|
||||
Instance_B1 : plan_cpu [inherited] = 400
|
||||
Instance_B1 : plan_gpu [inherited] = 50
|
||||
Instance_B1 : ...
|
||||
```
|
||||
|
||||
Below is a brief description of all the tables (hyperlinked to `tar.gz` files to be placed under [./data](./data) folder); the details are deferred to their sections.
|
||||
|
||||
- [**pai_job_table**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_job_table.tar.gz): job launch information.
|
||||
- [**pai_task_table**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_task_table.tar.gz): task launch information.
|
||||
- [**pai_instance_table**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_instance_table.tar.gz): instance launch information.
|
||||
- [**pai_sensor_table**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_sensor_table.tar.gz): instance resource sensor information.
|
||||
- [**pai_group_tag_table**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_group_tag_table.tar.gz): instance semantic information.
|
||||
- [**pai_machine_spec**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_spec.tar.gz): machine specification.
|
||||
- [**pai_machine_metric**](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_metric.tar.gz): machine resource metrics with respect to the instance.
|
||||
|
||||
(Alternative download links: [data repo on GitHub](https://github.com/qzweng/clusterdata-cluster-trace-gpu-v2020-data))
|
||||
|
||||
**Checksum** (replacing `sha256sum` with `shasum` for macOS users)
|
||||
```bash
|
||||
$sha256sum -c << EOF
|
||||
722fef30b7fb7aa50dabd79155614b5423a9d65cf45a9b26c590d57725423a14 pai_group_tag_table.tar.gz
|
||||
1bf1e423a7ce3f8d086699801c362fd56a7182abdb234139e5ebbed97995ca06 pai_instance_table.tar.gz
|
||||
5aad7f7caac501136d14ed6a48e40546f825d7b0617a3a4f337e2348fe0a6cb0 pai_job_table.tar.gz
|
||||
53ad917193d3b1dd0f3055e723148b1f36c2f81789b014ea2930a7875892eef5 pai_machine_metric.tar.gz
|
||||
cc0d38a4045af1b1af8179de8b1b54b1ddd995e6160d6d061a6b1000f1276c2d pai_machine_spec.tar.gz
|
||||
9a0b82e8bdf3949281e4ba1423d9b4b34847e52799eecb138966de46da69c7a0 pai_sensor_table.tar.gz
|
||||
cd1d6dc3215d2a8607ccf6b6dd952b5db776df86926c73259fea7c1499ac40e5 pai_task_table.tar.gz
|
||||
320470fa8f77182bc6052c95ff359b75542c748e9516b3d8148a6e2bafc1798d pai_group_tag_table.header
|
||||
51f2585178eaaf30ad8cc88c50ec9243b0f7d9603603ab7602efccb7aa34f6b0 pai_instance_table.header
|
||||
3ac33aefab9a4d81338794fa145fe280594a379444961a9c639f00181c508567 pai_job_table.header
|
||||
300342613103c6192490f8990958903d837053c13adbc6c094d677074fbbb7ad pai_machine_metric.header
|
||||
4c9ea25914ff3f0b73be9ff223fb91002e787138165abd3f584ba37281448e1c pai_machine_spec.header
|
||||
d3e26eb31fb3b833821373b487266312151a3bfb2f18fc62017a438546cc1362 pai_sensor_table.header
|
||||
978bbaabfc8695874c605c01c144b2977f611ceca73aeb72189988cdfbfb0a9c pai_task_table.header
|
||||
EOF
|
||||
```
|
||||
|
||||
**Size of file**
|
||||
```bash
|
||||
$ls -sh1 *.tar.gz
|
||||
53M pai_group_tag_table.tar.gz
|
||||
663M pai_instance_table.tar.gz
|
||||
60M pai_job_table.tar.gz
|
||||
198M pai_machine_metric.tar.gz
|
||||
32K pai_machine_spec.tar.gz
|
||||
388M pai_sensor_table.tar.gz
|
||||
34M pai_task_table.tar.gz
|
||||
```
|
||||
|
||||
**Usage**
|
||||
```bash
|
||||
$for file in `ls *.tar.gz`; do tar -xzf $file; done
|
||||
$sha256sum -c << EOF
|
||||
d23fcb56d88b2976fda4a62708d77b2a8c98635c4b443f100ed5553d4cd07fbe pai_group_tag_table.csv
|
||||
b179f7d7e0927a6663d719b728c92640447c5a0fc3c6e4edff31e7207ab9dd17 pai_instance_table.csv
|
||||
379ecb3becaba347f44a53bf7eb53e54b185221b2a0338a3f828828d269ba96c pai_job_table.csv
|
||||
8b6592daab0373ef7fec730198b4df5891b1722c2ab4c7016249166e131402d3 pai_machine_metric.csv
|
||||
251e9e33649155a6a935fdc1dc0e90cd4698206936b60bd8fce60ff2ccd51439 pai_machine_spec.csv
|
||||
12dd9929b70f3efe18d1279d9873e3a519c64f431613b23b059ed8c46a376dd7 pai_sensor_table.csv
|
||||
6954802b457305f8a9e480ef97c40060baee59649fd3adc62c5a1e048aa058de pai_task_table.csv
|
||||
EOF
|
||||
```
|
||||
|
||||
## pai_job_table
|
||||
**job launch information.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:-----------|:-------------------------------------------------------------|
|
||||
| job_name | 4b3f04b66a525d2df903eb16 |
|
||||
| inst_id | 8cb3bec23d14dbde320b6613452e768cbbf35b8bd64ee28fcceb77d3c47d |
|
||||
| user | 58540f191766 |
|
||||
| status | Terminated |
|
||||
| start_time | 4550879.0 |
|
||||
| end_time | 4551416.0 |
|
||||
|
||||
- `job_name`: name of users' submit jobs. It has been desensitized to protect users' privacy (similar to `user_name`, `worker_name`, `inst_name`, etc. below).
|
||||
- `inst_id`: please treat or revise it as `job_id`, since each `job_name` corresponds to one `inst_id`. It can be joined with `inst_id` in [pai_sensor_table](#pai_sensor_table) and [pai_group_tag_table](#pai_group_tag_table).
|
||||
- `user`: user name.
|
||||
- `status`: job status, including 'Running', 'Terminated', 'Failed', 'Waiting'; only 'Terminated' tasks are successful.
|
||||
- `start_time`: timestamp of job submission time.
|
||||
- `end_time`: timestamp of job completion time.
|
||||
|
||||
**Note of `time`**: Both `start_time` and `end_time` are in *seconds* and have been deducted by a constant number for desensitization. Still, if translated to Unix time in UTC+8 timezone ("Asia/Shanghai"), they will have the same time of the day (e.g., "08:59:59 UTC+8") and the same day of the week (e.g., "Sunday") as the original traces, while having fake dates, months, and years.
|
||||
|
||||
|
||||
|
||||
## pai_task_table
|
||||
|
||||
**task launch information.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:-----------|:-------------------------|
|
||||
| job_name | 4f057009a5d481acec67b088 |
|
||||
| task_name | tensorflow |
|
||||
| inst_num | 1.0 |
|
||||
| status | Terminated |
|
||||
| start_time | 1739162.0 |
|
||||
| end_time | 1739200.0 |
|
||||
| plan_cpu | 600.0 |
|
||||
| plan_mem | 29.296875 |
|
||||
| plan_gpu | 50.0 |
|
||||
| gpu_type | MISC |
|
||||
|
||||
- `job_name`: job name; same as the entry in [pai_job_table](#pai_job_table).
|
||||
- `task_name`: most jobs have only one task, but some may launch multiple tasks of different names (roles), e.g., `ps`, `worker`, `evaluator`.
|
||||
- `inst_num`: number of instances launched by the task.
|
||||
- `status`: task status.
|
||||
- `start_time`: timestamp of task launch time. The gap between `job.start_time` and the earliest `task.start_time` in the job implies its wait time before launching (scheduling latency).
|
||||
- `end_time`: timestamp of task completion time.
|
||||
- `plan_cpu`: number of CPU cores requested in percentage (i.e., 600.0 is 6 vCPU cores) .
|
||||
- `plan_mem`: GB of main memory requested.
|
||||
- `plan_gpu`: number of GPUs requested in percentage (i.e., 50.0 is 50% GPU).
|
||||
- `gpu_type`: type of GPUs assigned to this task. `MISC` is short for "miscellaneous", indicating GPUs of older generations, e.g., NVIDIA Tesla K40m, K80, M60.
|
||||
|
||||
|
||||
|
||||
## pai_instance_table
|
||||
|
||||
**instance launch information.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:------------|:-------------------------------------------------------------|
|
||||
| job_name | af724763f4f5d0beef445849 |
|
||||
| task_name | worker |
|
||||
| inst_name | 0d39aa867a79c16eff67daa8f6248f09af8346b177c9e3e23645c48354a8 |
|
||||
| worker_name | 54dbcd2db287841c03d0639b2a93e783a090ea085348f8cdb8e603d8b96f |
|
||||
| inst_id | e387fbc18d80cc3c9ca4f1f13ff1d46778c9a25eaaeca2a95314fdf20d8e |
|
||||
| status | Terminated |
|
||||
| start_time | 2081336.0 |
|
||||
| end_time | 2083889.0 |
|
||||
| machine | 471dda9ed84965451e042145 |
|
||||
|
||||
- `job_name`: job name; same as the entry in [pai_job_table](#pai_job_table).
|
||||
- `task_name`: task name; same as the entry in [pai_task_table](#pai_task_table).
|
||||
- `inst_name`: name of instance in each task.
|
||||
- `worker_name`: information to distinguish instances; it is more detailed than `inst_name` and to be joined with `worker_name` in [pai_sensor_table](#pai_sensor_table) and [pai_machine_metric](#pai_machine_metric).
|
||||
- `inst_id`: please treat or revise it as `job_id`, since each `job_name` corresponds to one `inst_id`; same as the entry in [pai_job_table](#pai_job_table)
|
||||
- `status`: instance status.
|
||||
- `start_time`: timestamp of instance launch time.
|
||||
- `end_time`: timestamp of instance completion time.
|
||||
- `machine`: the name of machine that the instance resides on, to be joined with `machine` in [pai_machine_spec](##pai_machine_spec) and [pai_machine_metric](#pai_machine_metric).
|
||||
|
||||
|
||||
|
||||
## pai_sensor_table
|
||||
|
||||
**instance resource sensor information.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:----------------|:-------------------------------------------------------------|
|
||||
| job_name | a9449d475665e3bf0512520b |
|
||||
| task_name | worker |
|
||||
| worker_name | bcecec52225d6b4ae6bc724ce0269a02026195a364f54cf4850c2cca0054 |
|
||||
| inst_id | 589de47b56f88129837f506134b874e0356dc0931732a687bcf907fb8325 |
|
||||
| machine | 6884752e3565b15cafe14218 |
|
||||
| gpu_name | /dev/nvidia0 |
|
||||
| cpu_usage | 140.1451612903226 |
|
||||
| gpu_wrk_util | 16.0625 |
|
||||
| avg_mem | 1.4627511160714286 |
|
||||
| max_mem | 2.3935546875 |
|
||||
| avg_gpu_wrk_mem | 1.2446746826171875 |
|
||||
| max_gpu_wrk_mem | 2.3994140625 |
|
||||
| read | 21271328.384615384 |
|
||||
| write | 16376189.815384615 |
|
||||
| read_count | 2922.4461538461537 |
|
||||
| write_count | 3419.7846153846153 |
|
||||
|
||||
- `job_name`: job name; same as the entry in [pai_job_table](#pai_job_table).
|
||||
- `task_name`: task name; same as the entry in [pai_task_table](#pai_task_table).
|
||||
- `worker_name`: worker name; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `inst_id`: please treat or revise it as `job_id`, since each `job_name` corresponds to one `inst_id`. Same as the entry in [pai_job_table](#pai_job_table)
|
||||
- `machine`: machine name; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `gpu_name`: name of the GPU on that machine (not `gpu_type`).
|
||||
- `cpu_usage`: number of CPU cores used in percentage (i.e., 600.0 is 6 vCPU cores) (c.f. `plan_cpu` in [pai_task_table](#pai_task_table)).
|
||||
- `gpu_wrk_util`: number of GPUs used in percentage (i.e., 50.0 is 50% GPU) (c.f., `plan_gpu` in [pai_task_table](#pai_task_table)).
|
||||
- `avg_mem`: GB of main memory used (in average) (c.f., `plan_mem` in [pai_task_table](#pai_task_table)).
|
||||
- `max_mem`: GB of main memory used (maximum) (c.f., `plan_mem` in [pai_task_table](#pai_task_table)).
|
||||
- `avg_gpu_wrk_mem`: GB of GPU memory used (in average).
|
||||
- `max_gpu_wrk_mem`: GB of GPU memory used (maximum).
|
||||
- `read`: Bytes of network input.
|
||||
- `write`: Bytes of network output.
|
||||
- `read_count`: Number of times of network read input.
|
||||
- `write_count`: Number of times of network write output.
|
||||
|
||||
**Note of `sensor`**: all the sensor metrics (CPU, GPU, Memory, I/O) in this table are collected for each *instance* (indexed by `worker_name`) but *not task*, taking the average of all data in the instance's lifetime (except for `max_mem` and `max_gpu_wrk_mem` being the maximum).
|
||||
|
||||
|
||||
|
||||
## pai_group_tag_table
|
||||
|
||||
**instance semantic information.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:--------------|:-------------------------------------------------------------|
|
||||
| inst_id | f7f6218cb5cb82e00b85476691d15d5055c143a351396d8f81737421dbd6 |
|
||||
| user | d2d3b77d342e |
|
||||
| gpu_type_spec | V100M32 |
|
||||
| group | fbeb14d671c629b6e82bee889fe4bb4c |
|
||||
| workload | nmt |
|
||||
|
||||
- `inst_id`: please treat or revise it as `job_id`, since each `job_name` corresponds to one `inst_id`. Same as the entry in [pai_job_table](#pai_job_table)
|
||||
- `user`: user name; same as the entry in [pai_job_table](#pai_job_table).
|
||||
- `gpu_type_spec`: being empty if the instance does not specify GPU type requirements, else being one of the `gpu_type` in [pai_task_table](#pai_task_table).
|
||||
- `group`: a semantic tag that indicates some instances have similar customized inputs, e.g., entry scripts, command-line parameters, data sources and sinks; consequently, instances with the same group tag are considered as repeated instances. Please refer to the trace analysis paper for detailed discussion.
|
||||
- `workload`: we study some Deep Learning tasks by investigating their customized inputs (mentioned above) and record their workload in this field; around 9% instances have this tag, including `graphlearn`, `ctr`, `bert`, etc.
|
||||
|
||||
|
||||
|
||||
## pai_machine_spec
|
||||
|
||||
**machine specification.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:---------|:-------------------------|
|
||||
| machine | 82e5af3cd5c4af3f56c62c54 |
|
||||
| gpu_type | T4 |
|
||||
| cap_cpu | 96 |
|
||||
| cap_mem | 512 |
|
||||
| cap_gpu | 2 |
|
||||
|
||||
- `machine`: machine name; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `gpu_type`: GPU type; same as the entry in [pai_task_table](#pai_task_table).
|
||||
- `cap_cpu`: CPU capacity; number of CPU cores in the machine.
|
||||
- `cap_mem`: memory capacity; GB of main memory capacity in the machine.
|
||||
- `cap_gpu`: GPU capacity; number of GPU in the machine.
|
||||
|
||||
|
||||
|
||||
## pai_machine_metric
|
||||
|
||||
**machine resource metrics with respect to the instance.**
|
||||
|
||||
| Columns | Example Entry |
|
||||
|:--------------------|:-------------------------------------------------------------|
|
||||
| worker_name | b739d0d058e0db100aaf47e48a4d61320c95c2f5a334a8262d5e830d849c |
|
||||
| machine | 74e1c8457b01c76b314b22bb |
|
||||
| start_time | 6150435 |
|
||||
| end_time | 6150689 |
|
||||
| machine_cpu_iowait | 0.0028667003281999497 |
|
||||
| machine_cpu_kernel | 3.583656890012642 |
|
||||
| machine_cpu_usr | 14.928745108999438 |
|
||||
| machine_gpu | 87.82875849911859 |
|
||||
| machine_load_1 | 18.298592909228066 |
|
||||
| machine_net_receive | 111649584.57135652 |
|
||||
| machine_num_worker | 5.053068410462776 |
|
||||
| machine_cpu | 18.515268699340282 |
|
||||
|
||||
- `worker_name`: worker name; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `machine`: machine name; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `start_time`: timestamp of instance launch time; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `end_time`: timestamp of instance completion; same as the entry in [pai_instance_table](#pai_instance_table).
|
||||
- `machine_cpu_iowait` : machine-level metrics of CPU I/O wait.
|
||||
- `machine_cpu_kernel` : machine-level metrics of CPU kernel usage.
|
||||
- `machine_cpu_usr` : machine-level metrics of CPU user usage.
|
||||
- `machine_gpu` : machine-level metrics of GPU utilization.
|
||||
- `machine_load_1` : machine-level metrics of 1-min load average.
|
||||
- `machine_net_receive` : machine-level metrics of network received bytes.
|
||||
- `machine_num_worker` : machine-level metrics of number of co-located instances (workers).
|
||||
- `machine_cpu` : machine-level metrics of CPU overall usage.
|
||||
|
||||
**Note of `machine_` metrics**: these metrics are machine-level metrics, taking average of the sensor data during the instance's (indexed by `worker_name`) lifetime.
|
||||
|
||||
|
||||
|
||||
# 3 Demo of Data Analysis
|
||||
|
||||
The displayed [Jupyter notebook](./analysis/analysis.ipynb) and the underlying [utility script](./analysis/utils.py) can be found under the [analysis](./analysis) directory.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
|
|
@ -1,261 +0,0 @@
|
|||
import os
|
||||
import datetime
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import seaborn as sns
|
||||
import matplotlib
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
########### Data Constants ###########
|
||||
DATA_DIR = '../data/'
|
||||
if not os.access('/tmp/figures', os.F_OK):
|
||||
os.mkdir('/tmp/figures')
|
||||
if not os.access('/tmp/figures', os.W_OK):
|
||||
print('Cannot write to /tmp/figures, please fix it.')
|
||||
exit()
|
||||
else:
|
||||
print('figures saved to /tmp/figures')
|
||||
|
||||
########### Prepare Functions ###########
|
||||
def get_df(file, header=None):
|
||||
df = pd.read_csv(file, header=None)
|
||||
# df.columns = DF_HEADER.get(key, df.columns)
|
||||
df.columns = pd.read_csv("{}.header".format(file.split('.csv')[0])).columns if header is None else header
|
||||
return df
|
||||
|
||||
def load_all_df():
|
||||
dfj = get_df(DATA_DIR + 'pai_job_table.csv')
|
||||
dft = get_df(DATA_DIR + 'pai_task_table.csv')
|
||||
dfi = get_df(DATA_DIR + 'pai_instance_table.csv')
|
||||
dfs = get_df(DATA_DIR + 'pai_sensor_table.csv')
|
||||
dfg = get_df(DATA_DIR + 'pai_group_tag_table.csv')
|
||||
dfp = get_df(DATA_DIR + 'pai_machine_spec.csv')
|
||||
dfm = get_df(DATA_DIR + 'pai_machine_metric.csv')
|
||||
return dfj,dft,dfi,dfs,dfg,dfp,dfm
|
||||
|
||||
def get_dfiw(dfi):
|
||||
dfiw = dfi.sort_values(['status','start_time','end_time'])
|
||||
dfiw.drop_duplicates(subset=['worker_name'], keep='last', inplace=True)
|
||||
dfiw.dropna(subset=['worker_name'], inplace=True)
|
||||
dfiw['runtime'] = dfiw[(dfiw.start_time>0)&(dfiw.end_time>0)]['end_time'] \
|
||||
- dfiw[(dfiw.start_time>0)&(dfiw.end_time>0)]['start_time']
|
||||
dfiw.loc[dfiw.start_time==0, 'start_time'] = np.nan
|
||||
dfiw.loc[dfiw.start_time==0, 'end_time'] = np.nan
|
||||
return dfiw
|
||||
|
||||
def get_dfw(dfi, dft, dfg):
|
||||
dfw = get_dfiw(dfi)
|
||||
dfw['start_date']=dfw.start_time.apply(pd.Timestamp, unit='s', tz='Asia/Shanghai')
|
||||
print('dfi + dft ...')
|
||||
dfw = dfw.merge(dft, on=['job_name','task_name'], how='left', suffixes=['', '_t'])
|
||||
print('dfi + dft + dfg ...')
|
||||
dfw = dfw.merge(dfg, on='inst_id', how='left') # reserve NaN ones by how='left'
|
||||
dfw.loc[dfw.group.isnull(),'group'] = dfw.loc[dfw.group.isnull(), 'user'] # fill group==NaN ones with user
|
||||
return dfw
|
||||
|
||||
def get_dfia(dfi):
|
||||
dfi_s = dfi[dfi.start_time > 0][['job_name','task_name','start_time']].groupby(['job_name','task_name']).min() # start_time
|
||||
dfi_e = dfi[dfi.end_time > 0][['job_name','task_name','end_time']].groupby(['job_name','task_name']).max() # end_time
|
||||
dfi_m = dfi[(dfi.start_time > 0) & (dfi.end_time > 0)][['job_name','task_name','end_time','start_time']]
|
||||
dfi_m['runtime'] = dfi_m.end_time-dfi_m.start_time
|
||||
dfi_m = dfi_m.groupby(['job_name','task_name']).mean()[['runtime']].reset_index() # runtime
|
||||
dfi_u = dfi[['job_name','task_name','status']].drop_duplicates().groupby(['job_name','task_name']).max() # status
|
||||
dfia = dfi_u
|
||||
for df in [dfi_s, dfi_e, dfi_m]:
|
||||
dfia = dfia.merge(df, on=['job_name','task_name'], how='left')
|
||||
return dfia
|
||||
|
||||
def get_dfa(dft, dfj, dfi, dfg):
|
||||
print('dft + dfj ...')
|
||||
dfa = dft.merge(dfj, on=['job_name'], suffixes = ['','_j'])
|
||||
dfa.loc[dfa.start_time==0, 'start_time'] = np.nan
|
||||
dfa.loc[dfa.start_time==0, 'end_time'] = np.nan
|
||||
dfa['runtime'] = dfa.end_time - dfa.start_time
|
||||
print('dft + dfj + dfi ...')
|
||||
dfia = get_dfia(dfi)
|
||||
dfa = dfa.merge(dfia, on=['job_name','task_name'], suffixes=['','_i'])
|
||||
dfa['duration_min'] = dfa.runtime_i / 60 # duration of instances
|
||||
dfa['wait_time'] = dfa.start_time_i - dfa.start_time # task wait time
|
||||
dfa['start_date']=dfa.start_time.apply(pd.Timestamp, unit='s', tz='Asia/Shanghai') # task start time
|
||||
# dfa = dfa[dfa.status=='Terminated']
|
||||
print('dft + dfj + dfi + dfg ...')
|
||||
dfa = dfa.merge(dfg[[x for x in dfg.columns if x != 'user']], on='inst_id', how='left') # reserve NaN ones by how='left'
|
||||
dfa.loc[dfa.group.isnull(),'group'] = dfa.loc[dfa.group.isnull(), 'user'] # fill group==NaN ones with user
|
||||
return dfa
|
||||
|
||||
def get_dfwitm(dfwit, csv_file='intermediate_data/machine_metric_shennong_machine_all.csv'):
|
||||
res_df = pd.read_csv(csv_file, index_col=0)
|
||||
dfwitm = dfwit.merge(res_df.loc[:, ~res_df.columns.isin(['start_time','end_time','machine'])], on='worker_name', how='left')
|
||||
return dfwitm
|
||||
|
||||
########### Plot Functions ###########
|
||||
linestyle_list = [
|
||||
('solid', 'solid'), # Same as (0, ()) or '-'
|
||||
('dotted', 'dotted'), # Same as (0, (1, 1)) or '.'
|
||||
('dashed', 'dashed'), # Same as '--'
|
||||
('dashdot', 'dashdot'), # Same as '-.'
|
||||
('densely dashdotdotted', (0, (3, 1, 1, 1, 1, 1))),
|
||||
('densely dashdotted', (0, (3, 1, 1, 1))),
|
||||
('densely dotted', (0, (1, 1))),
|
||||
('densely dashed', (0, (5, 1))),
|
||||
('dashdotdotted', (0, (3, 5, 1, 5, 1, 5))),
|
||||
('loosely dashed', (0, (5, 10))),
|
||||
('loosely dashdotted', (0, (3, 10, 1, 10))),
|
||||
('loosely dashdotdotted', (0, (3, 10, 1, 10, 1, 10))),
|
||||
('loosely dotted', (0, (1, 10))),
|
||||
('dashed', (0, (5, 5))),
|
||||
('dashdotted', (0, (3, 5, 1, 5))),
|
||||
('dotted', (0, (1, 1))),
|
||||
]
|
||||
|
||||
def get_cdf(data, inverse=False):
|
||||
sorted_data = sorted(data)
|
||||
p = 100. * np.arange(len(sorted_data))/(len(sorted_data)-1)
|
||||
p = 100. - p if inverse else p # CCDF
|
||||
return sorted_data, p
|
||||
|
||||
def plot_data_cdf(data, inverse=False, datalabel=None, xlabel=None, title=None, xlog=False, xlim=None, ylog=False, xticks=None, figsize=(4,3), dpi=120, savefig=None, ylabel=None):
|
||||
plt.figure(figsize=figsize, dpi=dpi)
|
||||
if type(data) == pd.DataFrame:
|
||||
data.dropna(inplace=True)
|
||||
x, y = get_cdf(data, inverse)
|
||||
plt.plot(x, y, label=datalabel, color='green', linestyle='-')
|
||||
if datalabel is not None: plt.legend(loc='lower right')
|
||||
if xlog: plt.xscale('log')
|
||||
if ylog: plt.yscale('log')
|
||||
if xlim is not None: plt.xlim(xlim)
|
||||
plt.ylim(0, 100)
|
||||
if xlabel is not None: plt.xlabel(xlabel)
|
||||
plt.ylabel(ylabel) if ylabel is not None else plt.ylabel('CCDF') if inverse is True else plt.ylabel('CDF')
|
||||
if title is not None: plt.title(title)
|
||||
if xticks is not None: plt.xticks(xticks)
|
||||
plt.grid(alpha=.3, linestyle='--')
|
||||
if savefig is not None:
|
||||
plt.savefig('/tmp/figures/{}.pdf'.format(savefig),bbox_inches='tight')
|
||||
else:
|
||||
plt.show()
|
||||
|
||||
def plot_data_cdfs(data, datalabel=None, inverse=False, xlabel=None, title=None, xlog=False, ylog=False, xticks=None, figsize=(4,3), dpi=120, xlim=None, ylim=None, ylabel=None, yticks=None, savefig=None, loc='best', fontsize=None):
|
||||
plt.figure(figsize=figsize, dpi=dpi)
|
||||
for i, d in enumerate(data):
|
||||
if type(data) == pd.DataFrame:
|
||||
d.dropna(inplace=True)
|
||||
x, y = get_cdf(d, inverse)
|
||||
label = datalabel[i] if datalabel is not None else None
|
||||
plt.plot(x, y, label=label, linestyle=linestyle_list[i % len(linestyle_list)][1])
|
||||
if datalabel is not None: plt.legend(loc=loc, fontsize=fontsize)
|
||||
if xlog: plt.xscale('log')
|
||||
if ylog: plt.yscale('log')
|
||||
plt.ylim(0, 100) if ylim is None else plt.ylim(ylim)
|
||||
if xlim is not None: plt.xlim(xlim)
|
||||
if xlabel is not None: plt.xlabel(xlabel)
|
||||
if ylabel is None:
|
||||
plt.ylabel('CCDF') if inverse is True else plt.ylabel('CDF')
|
||||
else:
|
||||
plt.ylabel(ylabel)
|
||||
if title is not None: plt.title(title)
|
||||
if xticks is not None: plt.xticks(xticks)
|
||||
if yticks is not None: plt.yticks(yticks)
|
||||
plt.grid(alpha=.3, linestyle='--')
|
||||
if savefig is not None:
|
||||
plt.savefig('/tmp/figures/{}.pdf'.format(savefig),bbox_inches='tight')
|
||||
else:
|
||||
plt.show()
|
||||
|
||||
def draw_bar_plot(odf, col, figsize=(4,4), dpi=120, portion=False, title=None, limit=30):
|
||||
dfout=odf.reset_index().groupby(col).count()[['index']].sort_values('index', ascending=False).head(limit)
|
||||
dfout['portion'] = 100 * dfout['index'] / dfout['index'].sum()
|
||||
plt.figure(figsize=figsize, dpi=dpi)
|
||||
if portion:
|
||||
plt.barh(y=dfout.index, width=dfout['portion'])
|
||||
plt.xlabel('Percentage (total: %.2f)'%(dfout['index'].sum()))
|
||||
else:
|
||||
plt.barh(y=dfout.index, width=dfout['index'])
|
||||
plt.grid(alpha=.3, linestyle='--')
|
||||
return dfout
|
||||
|
||||
########### Process Functions ###########
|
||||
|
||||
def get_inst_task_num_ratio(dfa, inst_num_list=[2, 8, 20, 64, 100, 256, 512]):
|
||||
total_num_task, total_num_inst = len(dfa), sum(dfa['inst_num'])
|
||||
data_df = []
|
||||
for i in inst_num_list:
|
||||
temp_df = dfa[dfa['inst_num'] >= i]
|
||||
task_num_ratio = len(temp_df) / total_num_task
|
||||
inst_num_ratio = sum(temp_df['inst_num']) / total_num_inst
|
||||
data_df.append([task_num_ratio, inst_num_ratio])
|
||||
out_df = pd.DataFrame(data_df, columns=['num_task_ratio','num_inst_ratio'])
|
||||
out_df = out_df.T.rename(columns=dict(zip(range(len(inst_num_list)), inst_num_list)))
|
||||
return out_df
|
||||
|
||||
|
||||
def add_hour_date(df):
|
||||
if 'start_date' not in df:
|
||||
if 'start_time_t' in df:
|
||||
target_col = 'start_time_t'
|
||||
elif 'start_time' in df:
|
||||
target_col = 'start_time'
|
||||
else:
|
||||
print('start_time, start_time_t, dayofyear unfound in df')
|
||||
return None
|
||||
df['start_date'] = df[target_col].apply(lambda x: pd.Timestamp(x, unit='s', tz='Asia/Shanghai'))
|
||||
if 'date' not in df:
|
||||
df['date'] = df['start_date'].apply(lambda x: x.date())
|
||||
if 'hour' not in df:
|
||||
df['hour'] = df['start_date'].apply(lambda x: x.hour)
|
||||
return df
|
||||
|
||||
def get_hourly_task_request(df): # df = dftjkix
|
||||
sum_df_list = []
|
||||
df = add_hour_date(df.copy())
|
||||
# for day in sorted(df.dayofyear.unique()):
|
||||
for date in sorted(df.date.unique()):
|
||||
# tempdf = df[df.dayofyear==day]
|
||||
tempdf = df[df.date==date]
|
||||
res_df = tempdf.groupby('hour').count()[['job_name']]
|
||||
res_df.rename(columns={'job_name':date}, inplace=True)
|
||||
sum_df_list.append(res_df.T)
|
||||
out_df = pd.DataFrame().append(sum_df_list)
|
||||
return out_df.dropna() # if a day contains hours of NaN, it is not a typical day
|
||||
|
||||
def get_hourly_task_resource_request(df, metrics='cpu'): # df = dftjkix
|
||||
sum_df_list = []
|
||||
df = add_hour_date(df)
|
||||
if metrics == 'cpu':
|
||||
df['plan_resource'] = df.plan_cpu.apply(lambda x: x/100)
|
||||
elif metrics == 'gpu':
|
||||
df['plan_resource'] = df.plan_gpu.apply(lambda x: x/100)
|
||||
elif metrics == 'mem':
|
||||
df['plan_resource'] = df.plan_mem.apply(lambda x: x/1000)
|
||||
else:
|
||||
exit()
|
||||
# for day in sorted(df.dayofyear.unique()):
|
||||
for date in sorted(df.date.unique()):
|
||||
# tempdf = df[df.dayofyear==day]
|
||||
tempdf = df[df.date==date]
|
||||
res_df = tempdf.groupby('hour').sum()[['plan_resource']]
|
||||
res_df.rename(columns={'job_name':date}, inplace=True)
|
||||
sum_df_list.append(res_df.T)
|
||||
out_df = pd.DataFrame().append(sum_df_list)
|
||||
return out_df.dropna() # if a day contains hours of NaN, it is not a typical day
|
||||
|
||||
def plan_minus_usg_over_cap_task(dfas):
|
||||
dfas['plan_gpu_minus_usage_over_capacity'] = (dfas['plan_gpu'] - dfas['gpu_wrk_util']) / (100 * dfas['cap_gpu'])
|
||||
dfas['plan_cpu_minus_usage_over_capacity'] = (dfas['plan_cpu'] - dfas['cpu_usage']) / (100 * dfas['cap_cpu'] )
|
||||
dfas['plan_mem_minus_usage_over_capacity'] = (dfas['plan_mem'] - dfas['avg_mem']) / dfas['cap_mem']
|
||||
|
||||
dfas_task = dfas.groupby(['job_name','task_name'])[['plan_gpu_minus_usage_over_capacity','plan_cpu_minus_usage_over_capacity','plan_mem_minus_usage_over_capacity']].mean()
|
||||
|
||||
pgu_datas, pgu_label, ugp_datas, ugp_label = [], [], [], []
|
||||
for device in ['cpu','gpu','mem']:
|
||||
apu = dfas_task[~dfas_task['plan_{}_minus_usage_over_capacity'.format(device)].isnull()]
|
||||
pgu = dfas_task[dfas_task['plan_{}_minus_usage_over_capacity'.format(device)] > 0]
|
||||
ugp = dfas_task[dfas_task['plan_{}_minus_usage_over_capacity'.format(device)] < 0]
|
||||
print("{}: plan > usage: {:.2f}%, plan < usage: {:.2f}%".format(
|
||||
device, 100 * len(pgu) / len(apu), 100 * len(ugp) / len(apu) ))
|
||||
pgu_label.append("{} {:.2f}%".format(device, 100 * len(pgu) / len(apu)))
|
||||
pgu_datas.append(pgu['plan_{}_minus_usage_over_capacity'.format(device)])
|
||||
ugp_label.append("{} {:.2f}%".format(device, 100 * len(ugp) / len(apu)))
|
||||
ugp_datas.append(-ugp['plan_{}_minus_usage_over_capacity'.format(device)])
|
||||
|
||||
return pgu_datas, ugp_datas, pgu_label, ugp_label
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
# Trace data
|
||||
|
||||
Due to the recent surge in pulls, this repository has been over the LFS quota. Alternatively, the traces can be downloaded from [Aliyun OSS](https://www.aliyun.com/product/oss) as follows (or running `bash download_data.sh`):
|
||||
|
||||
- [pai_group_tag_table.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_group_tag_table.tar.gz)
|
||||
- [pai_instance_table.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_instance_table.tar.gz)
|
||||
- [pai_job_table.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_job_table.tar.gz)
|
||||
- [pai_machine_metric.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_metric.tar.gz)
|
||||
- [pai_machine_spec.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_spec.tar.gz)
|
||||
- [pai_sensor_table.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_sensor_table.tar.gz)
|
||||
- [pai_task_table.tar.gz](https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_task_table.tar.gz)
|
||||
|
||||
(Backup: [data repo on GitHub](https://github.com/qzweng/clusterdata-cluster-trace-gpu-v2020-data))
|
||||
|
||||
Thank you for your support and understanding!
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_group_tag_table.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_instance_table.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_job_table.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_metric.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_machine_spec.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_sensor_table.tar.gz
|
||||
curl -O https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2020GPUTraces/pai_task_table.tar.gz
|
||||
|
|
@ -1 +0,0 @@
|
|||
inst_id,user,gpu_type_spec,group,workload
|
||||
|
|
@ -1 +0,0 @@
|
|||
job_name,task_name,inst_name,worker_name,inst_id,status,start_time,end_time,machine
|
||||
|
|
@ -1 +0,0 @@
|
|||
job_name,inst_id,user,status,start_time,end_time
|
||||
|
|
@ -1 +0,0 @@
|
|||
worker_name,machine,start_time,end_time,machine_cpu_iowait,machine_cpu_kernel,machine_cpu_usr,machine_gpu,machine_load_1,machine_net_receive,machine_num_worker,machine_cpu
|
||||
|
|
@ -1 +0,0 @@
|
|||
machine,gpu_type,cap_cpu,cap_mem,cap_gpu
|
||||
|
|
@ -1 +0,0 @@
|
|||
job_name,task_name,worker_name,inst_id,machine,gpu_name,cpu_usage,gpu_wrk_util,avg_mem,max_mem,avg_gpu_wrk_mem,max_gpu_wrk_mem,read,write,read_count,write_count
|
||||
|
|
@ -1 +0,0 @@
|
|||
job_name,task_name,inst_num,status,start_time,end_time,plan_cpu,plan_mem,plan_gpu,gpu_type
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 554 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 650 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 675 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.7 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 538 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 8.2 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 464 KiB |
|
|
@ -1,50 +0,0 @@
|
|||
# Mini-batch Time of Workers in Distributed Training Jobs on PAI
|
||||
|
||||
Motivated by the interest in the released traces on GPU workloads, we further release the mini-batch time of each worker in some distributed training jobs on the Alibaba Platform of AI (PAI).
|
||||
|
||||
"Mini-batch time": Deep learning training processes a mini-batch of data, typically between 8 and 512 samples, in each iteration. Usually, a training mini-batch can be divided into 1) forward pass, 2) backward pass, and 3) model parameters update [Xiao et al., 2020]. In distributed training jobs, multiple workers usually have the same mini-batch size (i.e., the number of samples) and perform the same operations on the data. The time they spent on processing their mini-batch of data is defined as mini-batch time.
|
||||
|
||||
Note that the mini-batch time has been normalized to 0-1000. The focus of this dataset is to reveal the diverse mini-batch time among different workers of the same training job rather than their precise duration. For detailed distribution of mini-batch time, please refer to Fig. 5(b) in the "AntMan" paper [Xiao et al., 2020].
|
||||
|
||||
> Xiao, W., Ren, S., Li, Y., Zhang, Y., Hou, P., Li, Z., ... & Jia, Y. (2020). AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. In Proc. USENIX OSDI.
|
||||
|
||||
## coefficient_variation.csv
|
||||
|
||||
Each row of data is one training job while columns are the iterations. Each value represents the coefficient of variation (CoV) calculated by all workers of a certain job in a certain iteration. The released data includes 100 training jobs that each have 10 workers.
|
||||
|
||||
Coefficient of variation is defined as: $c_v = \sigma/\mu$. (See [Wikipedia](https://en.wikipedia.org/wiki/Coefficient_of_variation)), where $\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^N(X_i-\mu)^2}$ is the standard deviation and $\mu$ is the mean value of all workers' mini-batch durations. Since the mean mini-batch time could vary widely among different jobs, we use CoV instead of standard deviation to capture this variation.
|
||||
|
||||
The following figure shows the CoV of 5 sampled jobs. The diversity of CoV can be observed among different jobs and different iterations during the training process.
|
||||
|
||||
|
||||
|
||||
|
||||
## job_1_norm.csv
|
||||
|
||||
- GPU request for each worker: 25% of one GPU
|
||||
- Observation: Straggler (worker 6) emerges in the middle of the training.
|
||||
|
||||
|
||||
|
||||
## job_2_norm.csv
|
||||
|
||||
- GPU request for each worker: 20% of one GPU
|
||||
- Observation: Although the variation seems stable during the training process, different workers have different mini-batch times.
|
||||
|
||||
|
||||
|
||||
|
||||
## job_3_norm.csv
|
||||
|
||||
- GPU request for each worker: 25% of one GPU
|
||||
- Observation: There could be multiple stragglers.
|
||||
|
||||
|
||||
|
||||
|
||||
## job_4_norm.csv
|
||||
|
||||
- GPU request for each worker: 50% of one GPU
|
||||
- Observation: There could be multiple stragglers. The mini-batch time may fluctuate.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 133 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 111 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 110 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 163 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 107 KiB |
|
|
@ -1,2 +0,0 @@
|
|||
.idea
|
||||
__pycache__
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
# Simulator of GPU Cluster Scheduling
|
||||
|
||||
## Code Overview
|
||||
### Structure
|
||||
```python
|
||||
simulator
|
||||
└── run_simulator.py
|
||||
└── simulator.py: Simulator()
|
||||
├── cluster.py: Cluster()
|
||||
│ ├── node.py: Node()
|
||||
│ └── job_history.py: JobHistory()
|
||||
└── scheduler.py: Scheduler()
|
||||
└── node.py: Node()
|
||||
```
|
||||
|
||||
### Key call path
|
||||
```python
|
||||
simulator
|
||||
└── run_simulator.py
|
||||
└── simulator.py
|
||||
└── simulator_go()
|
||||
├── init_go()
|
||||
│ ├── cluster = Cluster()
|
||||
│ └── simulator = Simulator()
|
||||
├── while not exit
|
||||
│ └── tic()
|
||||
│ ├── scheduler.preempt_job()
|
||||
│ └── scheduler.alloc_job()
|
||||
│ └── scheduler.alloc_job_sort()
|
||||
│ ├── SDF
|
||||
│ ├── FIFO
|
||||
│ └── ...
|
||||
└── exp_summary()
|
||||
```
|
||||
|
||||
### Key data structure
|
||||
- Job: an `OrderedDict` with keys of `user`, `duration`, estimated duration (`group_gpu_dur`), etc.
|
||||
- JobHistory: `self.user_job_stats` is a dict storing job statistics of each user, e.g., `num_job`, `dur_avg`.
|
||||
|
||||
## Usage
|
||||
```bash
|
||||
python3 run_simulator.py # compare multiple job scheduling policies (scheduler prefers load-balancing among nodes)
|
||||
# OR
|
||||
python3 run_simulator_fifo.py --pack # apply FIFO policy (scheduler prefers packing)
|
||||
```
|
||||
|
||||
### Output
|
||||
```bash
|
||||
log_file: ./logs/0228-pai_job_duration_estimate_100K.csv-99163-6500g_1n_h0_0p_3sn_0gt-1000ar-20000j-1x-42r.log
|
||||
==========
|
||||
20000_Jobs_repeated_1_times
|
||||
alloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime
|
||||
(SJF , LGF),5988.61,949.91,535706,20000,51.96
|
||||
(SJU , LGF),6278.33,1239.62,535706,20000,65.80
|
||||
(SJG , LGF),6071.01,1032.31,535706,20000,50.86
|
||||
(SJGG, LGF),6001.94,963.23,535706,20000,51.11
|
||||
(FIFO, LGF),7918.47,2879.77,535706,20000,110.53
|
||||
|
||||
# Sort by JCT
|
||||
(SJF , LGF),5988.61,949.91,535706,20000,51.96
|
||||
(SJGG, LGF),6001.94,963.23,535706,20000,51.11
|
||||
(SJG , LGF),6071.01,1032.31,535706,20000,50.86
|
||||
(SJU , LGF),6278.33,1239.62,535706,20000,65.80
|
||||
(FIFO, LGF),7918.47,2879.77,535706,20000,110.53
|
||||
|
||||
log_file: ./logs/0228-pai_job_duration_estimate_100K.csv-99163-6500g_1n_h0_0p_3sn_0gt-1000ar-20000j-1x-42r.log
|
||||
```
|
||||
### Headers
|
||||
- `alloc`: name of the allocation policy
|
||||
- `preempt`: name of the preemption policy, not used in this demo
|
||||
- `avg_jct`: average job completion time (jct)
|
||||
- `wait_time`: average job wait time (scheduling delay)
|
||||
- `makespan`: completion time when all jobs are finished
|
||||
- `jobs_done`: number of jobs done over all repeated experiments
|
||||
- `runtime`: wall clock time taken to run the experiments
|
||||
|
||||
### Allocation policies
|
||||
- `SJF`: 'Shortest Job First', SJF + Oracle, knowing each job's duration beforehand.
|
||||
- `SJU`: SJF + Duration Estimator using USER feature
|
||||
- `SJG`: SJF + Duration Estimator using GROUP, USER feature
|
||||
- `SJGG`: SJF + Duration Estimator using GROUP, USER, GPU feature
|
||||
- `FIFO`: FIFO, the default. Respect jobs' original arrival order, or random order in shuffled cases.
|
||||
|
||||
### Log file name explanation
|
||||
- `0228`: experiment date.
|
||||
- `pai_job_duration_estimate_100K.csv`: name of the traces file input.
|
||||
- `47996`: timestamp.
|
||||
- `6500g`: 6500 GPUs in the cluster.
|
||||
- `1n`: 1 Node (i.e., no topology, jobs can run on any GPU).
|
||||
- `h0`: heterogeneity: nil.
|
||||
- `0p`: resource dynamic pattern: nil (always 6500 GPUs).
|
||||
- `3sn`: scheduler policy: 3 is packing, 0 is load-balancing
|
||||
- `1000ar`: job arrival rate: 1000 jobs / minutes (60 seconds); `-1` is to use the original submit time.
|
||||
- `20000j`: 20000 jobs
|
||||
- `1x`: repeat 1 times
|
||||
- `42r`: random seed: 42
|
||||
|
|
@ -1,271 +0,0 @@
|
|||
from collections import OrderedDict
|
||||
from node import Node
|
||||
from utils import print_fn, _repr_job_preempt, _repr_job_done, large_job_pruning
|
||||
from job_history import JobHistory
|
||||
|
||||
|
||||
class Cluster:
|
||||
def __init__(self, node_list=None, num_nodes=None, num_gpus=20,
|
||||
num_cpus=20, pattern=1, period=124, job_list=None,
|
||||
random_seed=0, num_spare_node=None,
|
||||
export_cluster_util=False):
|
||||
if node_list is not None:
|
||||
node_list = node_list
|
||||
elif num_nodes is not None:
|
||||
node_list = [Node(id=i) for i in range(num_nodes)]
|
||||
else:
|
||||
node_list = [Node(id=0, num_gpus=num_gpus, num_cpus=num_cpus)]
|
||||
|
||||
temp_node_dict = dict()
|
||||
self.num_gpus, self.num_cpus = 0, 0
|
||||
for node in node_list:
|
||||
self.num_gpus += node.num_gpus
|
||||
self.num_cpus += node.num_cpus
|
||||
temp_node_dict[node.id] = node
|
||||
self.node_dict = OrderedDict(sorted(temp_node_dict.items(),
|
||||
key=lambda t: t[1].id))
|
||||
|
||||
self.cur_time = 0
|
||||
self.svc = {'num_gpu': 0, 'num_cpu': 0} # high-priority service
|
||||
self.svc_former_ratio = 0
|
||||
|
||||
# self.job_full_list = job_list # all jobs received from all times
|
||||
self.job_full_list = large_job_pruning(job_list, self.num_gpus, self.num_cpus)
|
||||
self.job_full_list.sort(key=lambda j: -j['submit_time'])
|
||||
self.job_list = []
|
||||
self.retrieve_job_from_full_list() # feed self.user_job_queue into self.job_list
|
||||
|
||||
self.job_history = JobHistory()
|
||||
|
||||
# Capacity changing pattern & period
|
||||
self.pattern = pattern
|
||||
self.period = period
|
||||
|
||||
# Spare specific node
|
||||
self.num_spare_node = num_spare_node
|
||||
self.spare_node_id = []
|
||||
if num_spare_node is not None:
|
||||
for i in range(num_spare_node):
|
||||
spare_node_index = random_seed % len(node_list)
|
||||
spare_node_id = node_list[spare_node_index].id
|
||||
while spare_node_id in self.spare_node_id:
|
||||
random_seed += 29741 # a random prime number
|
||||
spare_node_index = random_seed % len(node_list)
|
||||
spare_node_id = node_list[spare_node_index].id
|
||||
self.spare_node_id.append(spare_node_id) # indicate which node to spare
|
||||
random_seed += 29741 # a random prime number
|
||||
|
||||
self.export_cluster_util = export_cluster_util
|
||||
self.cluster_time = []
|
||||
self.cluster_cpu = []
|
||||
self.cluster_gpu = []
|
||||
self.idle_cluster_counter = 0
|
||||
|
||||
def retrieve_job_from_full_list(self):
|
||||
while len(self.job_full_list) > 0:
|
||||
job = self.job_full_list[-1]
|
||||
if job['submit_time'] <= self.cur_time:
|
||||
job = self.job_full_list.pop()
|
||||
self.job_list.append(job)
|
||||
else:
|
||||
return 0
|
||||
|
||||
def sorted_node_list(self):
|
||||
node_list = list(self.node_dict.values())
|
||||
node_list.sort(key=lambda n: n.id)
|
||||
return node_list
|
||||
|
||||
def tic_job(self, delta=1):
|
||||
# Unlike tic_svc(), it receives simulator's cur_time as its own cur_time
|
||||
# Here it returns a "cur_time" value to the simulator
|
||||
# If succeed: return cur_time >= 0
|
||||
# Else: return cur_time < 0 ==> exit_flag = 1
|
||||
self.cur_time += delta
|
||||
if self.export_cluster_util and self.cur_time % 10000 == 0:
|
||||
self.record_cluster_util()
|
||||
self.retrieve_job_from_full_list() # update self.job_list
|
||||
job_runn_list = self.job_runn_list
|
||||
if len(job_runn_list) > 0:
|
||||
for job in job_runn_list:
|
||||
job['on_time'] += delta
|
||||
job['progress'] = job['on_time'] * job['num_gpu']
|
||||
|
||||
# Job done logic
|
||||
if job['on_time'] >= job['duration']:
|
||||
over_tic_time = job['on_time'] - job['duration'] # only if delta > 1
|
||||
job['on_time'] -= over_tic_time
|
||||
job['progress'] -= over_tic_time * job['num_gpu']
|
||||
job['done'] = 1
|
||||
|
||||
host_node_id = job['node']
|
||||
host_node = self.node_dict.get(host_node_id)
|
||||
suc = host_node.release_job(job=job)
|
||||
assert suc
|
||||
|
||||
job['jct'] = self.cur_time - over_tic_time - job['submit_time'] # deduct submit_time
|
||||
|
||||
self.job_history.add_done_job(job)
|
||||
|
||||
print_fn("%sDONE: %s || %s" % (self.log_prefix, _repr_job_done(job), job))
|
||||
|
||||
return self.cur_time # exit_flag = 0, still going
|
||||
|
||||
# len(job_runn_list) <= 0,
|
||||
elif len(self.job_list) > 0: # empty cluster with job pending
|
||||
self.idle_cluster_counter += 1
|
||||
print_fn("%sIDLE cluster until jobs: %s" % (self.log_prefix, [_repr_job_preempt(e) for e in self.job_list]))
|
||||
|
||||
if self.idle_cluster_counter % 10000 == 0:
|
||||
print_fn('{} idle cluster: {}'.format(self.idle_cluster_counter, [_repr_job_preempt(e) for e in self.job_list]), level=2)
|
||||
return self.cur_time # exit_flag = 0, still going
|
||||
|
||||
elif len(self.job_full_list) > 0: # i.e., empty cluster waiting for jobs to come
|
||||
wake_time = self.job_full_list[-1]['submit_time'] - delta # the submit_time of the earliest job
|
||||
assert self.cur_time <= wake_time # if ==, i.e., the stride is unnecessary
|
||||
self.cur_time = wake_time
|
||||
return self.cur_time # exit_flag = 0, still going
|
||||
|
||||
else: # no running job, no pending job, no coming job => exit.
|
||||
return -1 # exit
|
||||
|
||||
def tic_svc(self, cur_time):
|
||||
self.cur_time = cur_time
|
||||
cap_ratio = self.get_cap_ratio(cur_time)
|
||||
svc_ratio = 1 - cap_ratio
|
||||
if self.svc_former_ratio != svc_ratio:
|
||||
self.svc_former_ratio = svc_ratio
|
||||
print_fn("%sService WAS:%s" % (self.log_prefix, str([n.__repr__() for n in self.node_list])))
|
||||
for node in self.node_list:
|
||||
if node.id in self.spare_node_id: # spare from service allocation
|
||||
continue
|
||||
node.set_svc_res_by_ratio(ratio=svc_ratio)
|
||||
print_fn("%sService NOW:%s" % (self.log_prefix, str([n.__repr__() for n in self.node_list])))
|
||||
|
||||
def replace_svc(self):
|
||||
# Migrating services or jobs for vacancies.
|
||||
raise NotImplementedError("Cluster replace service")
|
||||
|
||||
def display_capacity_pattern(self, max_time=200):
|
||||
for cur_time in range(max_time):
|
||||
cur_gpus, cur_cpus = self.get_capacity(cur_time)
|
||||
four_gpus, four_cpus = int(cur_gpus / 4), int(cur_cpus / 4)
|
||||
left_gpus, left_cpus = int(cur_gpus % 4), int(cur_cpus % 4)
|
||||
print("[%3s] G%3d |%s%s\n C%3d |%s%s" % (cur_time, cur_gpus, "####|" * four_gpus, "#" * left_gpus, cur_cpus, "xxxx|" * four_cpus, "x" * left_cpus ))
|
||||
|
||||
def display_capacity_pattern_csv(self, max_time=200):
|
||||
print("time,GPUs,CPUs")
|
||||
for cur_time in range(max_time):
|
||||
cur_gpus, cur_cpus = self.get_capacity(cur_time)
|
||||
# four_gpus, four_cpus = int(cur_gpus / 4), int(cur_cpus / 4)
|
||||
# left_gpus, left_cpus = int(cur_gpus % 4), int(cur_cpus % 4)
|
||||
print("%d,%d,%d" % (cur_time, cur_gpus, cur_cpus))
|
||||
|
||||
def get_capacity(self, time, num_spare_node=None):
|
||||
"""
|
||||
Only for display_capacity_pattern()
|
||||
:param time: cluster.cur_time, cluster.num_spare_node
|
||||
:return: [cur_gpus, cur_cpus]
|
||||
"""
|
||||
num_spare_node = self.num_spare_node if num_spare_node is None else num_spare_node
|
||||
ratio = self.get_cap_ratio(time)
|
||||
if num_spare_node is None:
|
||||
return [int(ratio * self.num_gpus), int(ratio * self.num_cpus)]
|
||||
else:
|
||||
if not self.spare_node_id:
|
||||
spare_node_id = list(range(num_spare_node))
|
||||
else:
|
||||
spare_node_id = self.spare_node_id
|
||||
g, c = 0, 0
|
||||
for node in self.node_list:
|
||||
if node.id in spare_node_id:
|
||||
g += node.num_gpus
|
||||
c += node.num_cpus
|
||||
else:
|
||||
g += node.num_gpus - int((1 - ratio) * node.num_gpus)
|
||||
c += node.num_cpus - int((1 - ratio) * node.num_cpus)
|
||||
assert g >= 0 and c >= 0
|
||||
return [g, c]
|
||||
|
||||
def get_cap_ratio(self, time, pattern=None, period=None):
|
||||
pattern = self.pattern if pattern is None else pattern
|
||||
period = self.period if period is None else period
|
||||
|
||||
pattern_ratio_dict = {
|
||||
0: {1:(0,1000)}, # always maximum capacity
|
||||
1: {1:(0, 62), 0.6:(62, 124)},
|
||||
2: {0.6:[(0, 10), (62, 124)], 1:(10, 62)},
|
||||
3: {1:(0, 20), 0.9:(20, 40), 0.8:(40, 60), 0.7:(60, 80), 0.6:(80, 100), 0.5:(100, 124)},
|
||||
4: {0.5:(0, 20), 0.6:(20, 40), 0.7:(40, 60), 0.8:(60, 80), 0.9:(80, 100)},
|
||||
5: {1:[(0, 10), (110, 124)], 0.9:[(10, 20),(100, 110)], 0.8:[(20, 30),(90, 100)], 0.7:[(30, 40),(80, 90)], 0.6:(40, 50), 0.5:(50, 70), 0.4:(70, 80)},
|
||||
6: {1:[(0, 20), (50, 60), (110, 124)], 0.6:(20, 50), 0.4:(60, 110)},
|
||||
7: {1:[(0, 20), (50, 60), (110, 124)], 0.9:(20, 50), 0.8:(60, 110)}
|
||||
} # { pattern1: {ratio1: [ (lower_bound1, upper_bound1), (lb2, ub2), ... ], ratio2: [...]}, pattern2: {...} }
|
||||
|
||||
t_mod_p = time % period
|
||||
ratio_dict = pattern_ratio_dict.get(pattern, {})
|
||||
for key, val in ratio_dict.items():
|
||||
if type(val) == tuple:
|
||||
val = [val] # becomes a list
|
||||
for bound in val:
|
||||
if bound[0] <= t_mod_p < bound[1]:
|
||||
return key
|
||||
return 1
|
||||
|
||||
def record_cluster_util(self):
|
||||
self.cluster_time.append(self.cur_time)
|
||||
self.cluster_cpu.append(self.job_cpus)
|
||||
self.cluster_gpu.append(self.job_gpus)
|
||||
|
||||
@property
|
||||
def node_list(self):
|
||||
return list(self.node_dict.values())
|
||||
|
||||
@property
|
||||
def cur_rsrc(self):
|
||||
return [self.cur_gpus, self.cur_cpus]
|
||||
|
||||
@property
|
||||
def cur_gpus(self):
|
||||
return self.num_gpus - self.svc_gpus
|
||||
|
||||
@property
|
||||
def cur_cpus(self):
|
||||
return self.num_cpus - self.svc_cpus
|
||||
|
||||
@property
|
||||
def job_runn_list(self):
|
||||
job_runn_list = []
|
||||
for node in self.node_list:
|
||||
job_runn_list.extend(node.job_runn_list)
|
||||
return job_runn_list
|
||||
|
||||
@property
|
||||
def svc_gpus(self):
|
||||
return sum([n.svc_gpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def svc_cpus(self):
|
||||
return sum([n.svc_cpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def idl_gpus(self):
|
||||
return sum([n.idl_gpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def idl_cpus(self):
|
||||
return sum([n.idl_cpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def job_gpus(self):
|
||||
return sum([n.job_gpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def job_cpus(self):
|
||||
return sum([n.job_cpus for n in self.node_list])
|
||||
|
||||
@property
|
||||
def log_prefix(self):
|
||||
if self.export_cluster_util is True: # add util export
|
||||
self.record_cluster_util()
|
||||
return "[%6s],[GPU,CPU]:[%7s,%8s]/[%7s,%8s]." % (self.cur_time, self.job_gpus, self.job_cpus, self.cur_gpus, self.cur_cpus)
|
||||
|
||||
|
|
@ -1,130 +0,0 @@
|
|||
from utils import add_user_round_robin_id
|
||||
import numpy as np
|
||||
|
||||
|
||||
class JobHistory:
|
||||
"""
|
||||
As a class to record run jobs as "job history"
|
||||
Inc.: 1. job_done_list
|
||||
2. user_job_stats
|
||||
3. visualization
|
||||
"""
|
||||
def __init__(self, job_done_list=None, window=10):
|
||||
self.job_done_list = list() if job_done_list is None else job_done_list.copy()
|
||||
self.user_job_stats = {}
|
||||
self.window = window
|
||||
self.job_window_list_dict = {}
|
||||
self.temp_user_stats = {}
|
||||
for job in self.job_done_list:
|
||||
self.stats_add_done_job(job) # Only stats recorded
|
||||
# No jct summary here
|
||||
|
||||
def alloc_job_sort(self, job_list, job_runn_list=None, metrics='dur_avg'):
|
||||
add_user_round_robin_id(job_list)
|
||||
# Early estimation: adding running jobs' on_time as sorting metrics
|
||||
if job_runn_list is not None:
|
||||
for job in job_runn_list:
|
||||
if job['user'] not in self.user_job_stats:
|
||||
self.temp_user_stats[job['user']] = job['on_time'] # take job's current on_time as duration estimation
|
||||
job_list.sort(key=lambda j: (self.get_job_metrics(j, metrics), j['user_rrid']))
|
||||
|
||||
def get_job_metrics(self, job, metrics):
|
||||
user = job['user']
|
||||
if user in self.user_job_stats:
|
||||
return self.user_job_stats[user][metrics]
|
||||
elif user in self.temp_user_stats:
|
||||
return self.temp_user_stats[user]
|
||||
else:
|
||||
return 0 # no history, no running job ==> duration = 0 ==> highest priority
|
||||
|
||||
def stats_add_done_job(self, job):
|
||||
window_list = [] # Could use deque for better performance
|
||||
user = job['user']
|
||||
dur = job['duration']
|
||||
if user not in self.user_job_stats:
|
||||
assert user not in self.job_window_list_dict
|
||||
self.job_window_list_dict[user] = [dur]
|
||||
|
||||
self.user_job_stats[user] = {
|
||||
'num_job': 1,
|
||||
'dur_avg': dur,
|
||||
'dur_mva': dur,
|
||||
'dur_har': min(1, 1 / dur) # Harmonic mean
|
||||
}
|
||||
else:
|
||||
job_win_list = self.job_window_list_dict[user]
|
||||
length = len(job_win_list)
|
||||
if length >= self.window:
|
||||
job_win_list.pop(0)
|
||||
job_win_list.append(dur)
|
||||
|
||||
dur_avg = self.user_job_stats[user]['dur_avg']
|
||||
dur_har = self.user_job_stats[user]['dur_har']
|
||||
num_job = self.user_job_stats[user]['num_job']
|
||||
dur_avg = dur_avg + (dur - dur_avg) / (num_job + 1)
|
||||
dur_har = dur_har + (min(1, 1/dur) - dur_har) / (num_job + 1)
|
||||
self.user_job_stats[user] = {
|
||||
'num_job': num_job + 1,
|
||||
'dur_avg': dur_avg,
|
||||
'dur_mva': sum(job_win_list)/len(job_win_list),
|
||||
'dur_har': dur_har
|
||||
}
|
||||
|
||||
def add_done_job(self, job):
|
||||
self.job_done_list_add_job(job)
|
||||
self.stats_add_done_job(job)
|
||||
# self.jct_summary_v += job['jct'] # No jct summary here
|
||||
|
||||
def job_done_list_add_job(self, job):
|
||||
""" form a more concise repr of job """
|
||||
job_dict = {}
|
||||
for k, v in job.items():
|
||||
if k in ['job_id', 'submit_time', 'duration', 'jct', 'wasted',
|
||||
'num_inst', 'num_cpu', 'num_gpu', 'node', 'user', 'gpu_type']:
|
||||
job_dict[k] = v
|
||||
self.job_done_list.append(job_dict)
|
||||
|
||||
@property
|
||||
def num_jobs_done(self):
|
||||
num_jobs_done = 0
|
||||
for stats in self.user_job_stats.values():
|
||||
num_jobs_done += stats['num_job']
|
||||
return num_jobs_done
|
||||
|
||||
# Not recommended using job_done_list
|
||||
@property
|
||||
def jct_summary(self):
|
||||
jct_summary = 0
|
||||
for job in self.job_done_list:
|
||||
jct_summary += job['jct']
|
||||
# assert self.jct_summary_v == jct_summary
|
||||
return jct_summary
|
||||
|
||||
@property
|
||||
def wait_time_summary(self):
|
||||
wait_time_summary = 0
|
||||
for job in self.job_done_list:
|
||||
wait_time_summary += job['jct'] - job['duration']
|
||||
return wait_time_summary
|
||||
|
||||
@property
|
||||
def wasted_summary(self):
|
||||
wasted_summary = 0
|
||||
for job in self.job_done_list:
|
||||
wasted_summary += job['wasted']
|
||||
return wasted_summary
|
||||
|
||||
def predict(self, user, metrics=None):
|
||||
"""
|
||||
metrics: in ['dur_avg', 'dur_har', 'dur_mva']
|
||||
"""
|
||||
|
||||
metrics = 'dur_avg' if metrics is None else metrics
|
||||
assert metrics in ['dur_avg', 'dur_har', 'dur_mva']
|
||||
if user not in self.user_job_stats:
|
||||
return 0 # No record => optimistic est. as 0.
|
||||
else:
|
||||
res = self.user_job_stats[user][metrics]
|
||||
if np.isnan(res):
|
||||
raise TypeError
|
||||
return res
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
==========
|
||||
20000_Jobs_repeated_1_times
|
||||
alloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime
|
||||
(SJF , LGF),5988.61,949.91,535706,20000,51.96
|
||||
(SJU , LGF),6278.33,1239.62,535706,20000,65.80
|
||||
(SJG , LGF),6071.01,1032.31,535706,20000,50.86
|
||||
(SJGG, LGF),6001.94,963.23,535706,20000,51.11
|
||||
(FIFO, LGF),7918.47,2879.77,535706,20000,110.53
|
||||
|
||||
# Sort by JCT
|
||||
alloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime
|
||||
(SJF , LGF),5988.61,949.91,535706,20000,51.96
|
||||
(SJGG, LGF),6001.94,963.23,535706,20000,51.11
|
||||
(SJG , LGF),6071.01,1032.31,535706,20000,50.86
|
||||
(SJU , LGF),6278.33,1239.62,535706,20000,65.80
|
||||
(FIFO, LGF),7918.47,2879.77,535706,20000,110.53
|
||||
|
|
@ -1,265 +0,0 @@
|
|||
from __future__ import absolute_import
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
from utils import print_fn
|
||||
|
||||
'''
|
||||
Class Node
|
||||
'''
|
||||
|
||||
|
||||
class Node(object):
|
||||
def __init__(self, id, num_gpus=8, num_cpus=96, mem=720,
|
||||
job_runn_list=None, gpu_type=0):
|
||||
self.id = id
|
||||
self.num_gpus = num_gpus * 100 # in %
|
||||
self.idl_gpus = self.num_gpus # in %
|
||||
self.svc_gpus = 0 # occupied by higher-priority services
|
||||
self.job_gpus = 0 # sum([j['num_gpu'] for j in self.job_runn_list])
|
||||
# num_gpus = svc_gpus + job_gpus + idl_gpus
|
||||
# = svc_gpus + cur_gpus
|
||||
|
||||
self.gpu_type = gpu_type
|
||||
|
||||
self.num_cpus = num_cpus * 100 # in %
|
||||
self.idl_cpus = self.num_cpus # in %
|
||||
self.svc_cpus = 0
|
||||
self.job_cpus = 0
|
||||
|
||||
self.mem = mem
|
||||
self.idl_mem = mem
|
||||
self.svc_mem = 0
|
||||
self.job_mem = 0
|
||||
|
||||
self.network_in = 0 # bw, or traffic amount
|
||||
self.network_out = 0
|
||||
|
||||
if job_runn_list is None:
|
||||
self.job_runn_list = list()
|
||||
else:
|
||||
self.job_runn_list = list(job_runn_list)
|
||||
|
||||
print_fn(' Node[%3d]:(%3d GPUs,%4d CPUs) %s' %
|
||||
(id, num_gpus, num_cpus, gpu_type))
|
||||
|
||||
@property
|
||||
def util_rate(self):
|
||||
cpus_util = 1 - self.idl_cpus / self.num_cpus
|
||||
if self.num_gpus > 0:
|
||||
gpus_util = 1 - self.idl_gpus / self.num_gpus
|
||||
util_rate = round(100 * (gpus_util + cpus_util) / 2)
|
||||
else:
|
||||
util_rate = round(100 * cpus_util)
|
||||
return util_rate
|
||||
|
||||
def __repr__(self):
|
||||
self.update_idl_gpus()
|
||||
self.update_idl_cpus()
|
||||
return 'N[%3d]: [(j %3d,i %3d)/%3d GPUs, (j %4d,i %4d)/%4d CPUs -- %3d Util.] %s' % (
|
||||
self.id, self.job_gpus, self.idl_gpus, self.num_gpus, self.job_cpus, self.idl_cpus, self.num_cpus,
|
||||
self.util_rate, self.gpu_type)
|
||||
|
||||
def check_rsrc(self):
|
||||
assert self.num_gpus == self.svc_gpus + self.idl_gpus + self.job_gpus
|
||||
assert self.num_cpus == self.svc_cpus + self.idl_cpus + self.job_cpus
|
||||
|
||||
def update_idl_gpus(self):
|
||||
self.idl_gpus = self.num_gpus - self.svc_gpus - self.job_gpus
|
||||
|
||||
def update_idl_cpus(self):
|
||||
self.idl_cpus = self.num_cpus - self.svc_cpus - self.job_cpus
|
||||
|
||||
'''alloc/release job'''
|
||||
|
||||
def alloc_job(self, job):
|
||||
if self.alloc_res(num_gpus=job['num_gpu'], num_cpus=job['num_cpu']):
|
||||
self.job_runn_list.append(job)
|
||||
self.job_gpus += job['num_gpu']
|
||||
self.job_cpus += job['num_cpu']
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
|
||||
def release_job(self, job):
|
||||
if self.release_res(num_gpus=job['num_gpu'], num_cpus=job['num_cpu']):
|
||||
self.job_runn_list.remove(job)
|
||||
self.job_gpus -= job['num_gpu']
|
||||
self.job_cpus -= job['num_cpu']
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
|
||||
'''alloc/release srv'''
|
||||
|
||||
def set_svc_res_by_ratio(self, ratio=0):
|
||||
self.svc_gpus = int(ratio * self.num_gpus)
|
||||
self.svc_cpus = int(ratio * self.num_cpus)
|
||||
self.update_idl_gpus()
|
||||
self.update_idl_cpus()
|
||||
|
||||
'''alloc/release resource'''
|
||||
|
||||
def alloc_res(self, num_gpus=0, num_cpus=0):
|
||||
# alloc job resource
|
||||
gpu = self.alloc_gpus(num_gpus)
|
||||
cpu = self.alloc_cpus(num_cpus)
|
||||
|
||||
if not cpu and not gpu:
|
||||
return False
|
||||
elif not cpu and gpu:
|
||||
self.release_gpus(num_gpus)
|
||||
return False
|
||||
elif cpu and not gpu:
|
||||
self.release_cpus(num_cpus)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def release_res(self, num_gpus, num_cpus):
|
||||
# input is gpu and cpu
|
||||
cpu = self.release_cpus(num_cpus)
|
||||
gpu = self.release_gpus(num_gpus)
|
||||
|
||||
return cpu and gpu
|
||||
|
||||
'''alloc/release resource with best efforts'''
|
||||
|
||||
def alloc_gpu_best_effort(self, num_gpus=0):
|
||||
"""return: num_gpus_left_to_alloc"""
|
||||
assert num_gpus >= 0
|
||||
|
||||
if num_gpus <= self.idl_gpus:
|
||||
self.svc_gpus += num_gpus
|
||||
self.idl_gpus -= num_gpus
|
||||
num_gpus = 0
|
||||
else:
|
||||
self.svc_gpus += self.idl_gpus
|
||||
num_gpus -= self.idl_gpus
|
||||
self.idl_gpus = 0
|
||||
return num_gpus
|
||||
|
||||
def alloc_cpu_best_effort(self, num_cpus=0):
|
||||
"""return: num_cpus_left_to_alloc"""
|
||||
assert num_cpus >= 0
|
||||
if num_cpus <= self.idl_cpus:
|
||||
self.svc_cpus += num_cpus
|
||||
self.idl_cpus -= num_cpus
|
||||
num_cpus = 0
|
||||
else:
|
||||
self.svc_cpus += self.idl_cpus
|
||||
num_cpus -= self.idl_cpus
|
||||
self.idl_cpus = 0
|
||||
return num_cpus
|
||||
|
||||
def release_gpu_best_effort(self, num_gpus=0):
|
||||
"""return: num_gpus_left_to_release"""
|
||||
assert num_gpus >= 0
|
||||
if num_gpus <= self.svc_gpus:
|
||||
self.idl_gpus += num_gpus
|
||||
self.svc_gpus -= num_gpus
|
||||
num_gpus = 0
|
||||
else:
|
||||
self.idl_gpus += self.svc_gpus
|
||||
num_gpus -= self.svc_gpus
|
||||
self.svc_gpus = 0
|
||||
return num_gpus
|
||||
|
||||
def release_cpu_best_effort(self, num_cpus=0):
|
||||
"""return: num_cpus_left_to_release"""
|
||||
assert num_cpus >= 0
|
||||
if num_cpus <= self.svc_cpus:
|
||||
self.idl_cpus += num_cpus
|
||||
self.svc_cpus -= num_cpus
|
||||
num_cpus = 0
|
||||
else:
|
||||
self.idl_cpus += self.svc_cpus
|
||||
num_cpus -= self.svc_cpus
|
||||
self.svc_cpus = 0
|
||||
return num_cpus
|
||||
|
||||
''' GPU '''
|
||||
|
||||
def get_idl_gpus(self):
|
||||
return self.idl_gpus
|
||||
|
||||
def alloc_gpus(self, num_gpus=0):
|
||||
'''
|
||||
If enough free gpus, allocate gpus
|
||||
Return: True, for success;
|
||||
False, for failure
|
||||
'''
|
||||
if num_gpus > self.idl_gpus:
|
||||
return False
|
||||
else:
|
||||
self.idl_gpus -= num_gpus
|
||||
return True
|
||||
|
||||
def release_gpus(self, num_gpus=0):
|
||||
'''
|
||||
release using gpus back to free list
|
||||
'''
|
||||
if self.idl_gpus + num_gpus > self.num_gpus:
|
||||
self.idl_gpus = self.num_gpus
|
||||
return False
|
||||
else:
|
||||
self.idl_gpus += num_gpus
|
||||
return True
|
||||
|
||||
''' CPU '''
|
||||
|
||||
def get_idl_cpus(self):
|
||||
return self.idl_cpus
|
||||
|
||||
def alloc_cpus(self, num_cpus=0):
|
||||
'''
|
||||
If enough free cpus, allocate gpus
|
||||
Return: True, for success;
|
||||
False, for failure
|
||||
'''
|
||||
if num_cpus > self.idl_cpus:
|
||||
return False
|
||||
else:
|
||||
self.idl_cpus -= num_cpus
|
||||
return True
|
||||
|
||||
def release_cpus(self, num_cpus=0):
|
||||
'''
|
||||
release using cpus back to free list
|
||||
'''
|
||||
if self.idl_cpus + num_cpus > self.num_cpus:
|
||||
self.idl_cpus = self.num_cpus
|
||||
return False
|
||||
else:
|
||||
self.idl_cpus += num_cpus
|
||||
return True
|
||||
|
||||
'''network'''
|
||||
|
||||
def add_network_load(self, in_load=0, out_load=0):
|
||||
self.network_in += in_load
|
||||
self.network_out += out_load
|
||||
self.network_in = round(self.network_in, 1)
|
||||
self.network_out = round(self.network_in, 1)
|
||||
|
||||
def release_network_load(self, in_load=0, out_load=0):
|
||||
self.network_in -= in_load
|
||||
self.network_out -= out_load
|
||||
self.network_in = round(self.network_in, 1)
|
||||
self.network_out = round(self.network_in, 1)
|
||||
|
||||
def set_network_load(self, in_load=0, out_load=0):
|
||||
self.network_in = in_load
|
||||
self.network_out = out_load
|
||||
self.network_in = round(self.network_in, 1)
|
||||
self.network_out = round(self.network_in, 1)
|
||||
|
||||
def init_node(self, num_gpus=0, num_cpus=0, mem=0):
|
||||
if num_gpus != 0:
|
||||
self.num_gpus = num_gpus
|
||||
self.idl_gpus = num_gpus
|
||||
if num_cpus != 0:
|
||||
self.num_cpus = num_cpus
|
||||
self.idl_cpus = num_cpus
|
||||
if mem != 0:
|
||||
self.mem = mem
|
||||
self.idl_mem = mem
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
numpy
|
||||
matplotlib
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
# usage: python3 run_simulator.py & date
|
||||
|
||||
from simulator import Simulator
|
||||
from utils import print_fn, ALLOC_POLICY_DICT, PREEMPT_POLICY_DICT
|
||||
import os
|
||||
import time
|
||||
import logging
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
DATE = "%02d%02d" % (time.localtime().tm_mon, time.localtime().tm_mday)
|
||||
|
||||
# INPUT TRACE FILE
|
||||
CSV_FILE_PATH = Path(__file__).parent / 'traces/pai/'
|
||||
DESCRIBE_FILE = None
|
||||
CSV_FILE = 'pai_job_duration_estimate_100K.csv'
|
||||
|
||||
parser = argparse.ArgumentParser(description='Simulator.')
|
||||
parser.add_argument("-r", "--arrival_rate", help="Arrival Rate", type=int, default=1000)
|
||||
parser.add_argument("-n", "--num_jobs", help="Num of Jobs", type=int, default=20000)
|
||||
parser.add_argument("-g", "--num_gpus", help="Num of GPUs", type=int, default=6500)
|
||||
parser.add_argument("-p", "--repeat", help='Repeat', type=int, default=1)
|
||||
parser.add_argument("-k", "--pack", dest='packing_policy', action='store_true')
|
||||
parser.add_argument("-b", "--balance", dest='packing_policy', action='store_false')
|
||||
parser.set_defaults(packing_policy=False)
|
||||
args = parser.parse_args()
|
||||
NUM_JOBS = args.num_jobs
|
||||
ARRIVAL_RATE = args.arrival_rate
|
||||
NUM_GPUS = args.num_gpus
|
||||
REPEAT = args.repeat
|
||||
SORT_NODE_POLICY = 0 if args.packing_policy is True else 3 # 0: packing, 3: max-min balancing.
|
||||
|
||||
MAX_TIME = int(1e9)
|
||||
|
||||
VERBOSE = 0
|
||||
# VERBOSE = 1
|
||||
# LOG_LEVEL = logging.DEBUG
|
||||
# LOG_LEVEL = logging.INFO
|
||||
LOG_LEVEL = logging.WARNING
|
||||
|
||||
NUM_NODES = 1
|
||||
NUM_CPUS = round(23.22 * NUM_GPUS) # 23.22 * num_gpus 156576/6742
|
||||
# HETERO = True # heterogeneous cluster
|
||||
HETERO = False
|
||||
PATTERN = 0 # Cluster capacity varying pattern
|
||||
|
||||
# GPU_TYPE_MATCHING = 1 # GPU type perfect match
|
||||
# GPU_TYPE_MATCHING = 2 # Only V100 cannot compromise
|
||||
GPU_TYPE_MATCHING = 0
|
||||
|
||||
EXPORT_JOB_STATS = False
|
||||
# EXPORT_JOB_STATS = True
|
||||
EXPORT_CLUSTER_UTIL = False
|
||||
# EXPORT_CLUSTER_UTIL = True
|
||||
|
||||
# RANDOM_SEED = random.randint(0, 100)
|
||||
RANDOM_SEED = 42
|
||||
NUM_SPARE_NODE = 0
|
||||
# SORT_BY_JCT = False
|
||||
SORT_BY_JCT = True
|
||||
|
||||
# Logging in directory
|
||||
LOG_DIR = Path(__file__).parent / 'logs'
|
||||
|
||||
comments = '%dg_%dn_h%d_%dp_%dsn_%dgt-%dar-%dj-%dx-%dr' % (NUM_GPUS, NUM_NODES, HETERO, PATTERN, SORT_NODE_POLICY, GPU_TYPE_MATCHING, ARRIVAL_RATE, NUM_JOBS, REPEAT, RANDOM_SEED)
|
||||
|
||||
log_time = int(time.time() % 100000)
|
||||
if not os.path.exists(LOG_DIR):
|
||||
os.makedirs(LOG_DIR)
|
||||
|
||||
log_file = LOG_DIR / ("%s-%s-%s-%s.log" % (DATE, CSV_FILE, log_time, comments))
|
||||
logging.basicConfig(level=LOG_LEVEL, format="%(message)s", filename=log_file, filemode='a')
|
||||
describe_file = CSV_FILE_PATH / DESCRIBE_FILE if DESCRIBE_FILE is not None else None
|
||||
|
||||
results_dict = {}
|
||||
num_jobs_dict = {}
|
||||
avg_jct_dict = {}
|
||||
makespan_dict = {}
|
||||
wait_time_dict = {}
|
||||
runtime_dict = {}
|
||||
|
||||
print("log_file: %s" % log_file)
|
||||
print_str = "==========\n%d_Jobs_repeated_%d_times\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime" % (NUM_JOBS, REPEAT)
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
for alloc_policy in [0, 1, 2, 4, 8]: # 0SDF, 1SJU, 2SJG, 4SJGG, 8FIFO (see utils.py)
|
||||
for preempt_policy in [2]: # 2LGF
|
||||
key = (alloc_policy, preempt_policy)
|
||||
print_key = "(%-4s,%4s)" % (ALLOC_POLICY_DICT.get(key[0]), PREEMPT_POLICY_DICT.get(key[1]))
|
||||
|
||||
# running
|
||||
start_time = time.time()
|
||||
print_fn("\n###### %s ######" % print_key)
|
||||
|
||||
simulator = Simulator(
|
||||
csv_file=CSV_FILE_PATH / CSV_FILE,
|
||||
alloc_policy=alloc_policy,
|
||||
preempt_policy=preempt_policy,
|
||||
sort_node_policy=SORT_NODE_POLICY,
|
||||
num_nodes=NUM_NODES,
|
||||
random_seed=RANDOM_SEED,
|
||||
max_time=MAX_TIME,
|
||||
num_spare_node=NUM_SPARE_NODE,
|
||||
pattern=PATTERN,
|
||||
hetero=HETERO,
|
||||
num_gpus=NUM_GPUS,
|
||||
num_cpus=NUM_CPUS,
|
||||
describe_file=describe_file,
|
||||
log_file=log_file,
|
||||
export_job_stats=EXPORT_JOB_STATS,
|
||||
export_cluster_util=EXPORT_CLUSTER_UTIL,
|
||||
arrival_rate=ARRIVAL_RATE,
|
||||
num_jobs_limit=NUM_JOBS,
|
||||
gpu_type_matching=GPU_TYPE_MATCHING,
|
||||
verbose=VERBOSE)
|
||||
results = simulator.simulator_go(repeat=REPEAT)
|
||||
|
||||
# post processing
|
||||
num_jobs, avg_jct, makespan, wait_time = 0, 0, 0, 0
|
||||
for item in results: # [num_jobs, avg_jct, makespan, [#alloc, alloc_time, #preempt, preempt_time]]
|
||||
num_jobs += item[0]
|
||||
avg_jct += item[1]
|
||||
wait_time += item[2]
|
||||
makespan += item[3]
|
||||
# key = (alloc_policy, preempt_policy)
|
||||
results_dict[key] = results
|
||||
num_jobs_dict[key] = num_jobs
|
||||
avg_jct_dict[key] = avg_jct / REPEAT
|
||||
makespan_dict[key] = makespan / REPEAT
|
||||
wait_time_dict[key] = wait_time / REPEAT
|
||||
runtime_dict[key] = time.time() - start_time
|
||||
|
||||
# print_fn("##### %s runtime: %.4f sec #####\n" % (print_key, runtime_dict[key]))
|
||||
|
||||
print_str = "%s,%.2f,%.2f,%.0f,%d,%.2f" % (print_key, avg_jct_dict[key], wait_time_dict[key], makespan_dict[key], num_jobs_dict[key], runtime_dict[key])
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
if SORT_BY_JCT:
|
||||
print("\n# Sort by JCT")
|
||||
print_fn("\n# Sort by JCT\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime", level=2)
|
||||
items = sorted(avg_jct_dict.items(), key=lambda d: d[1])
|
||||
else:
|
||||
print("\n# Summary")
|
||||
print_fn("\n# Summary\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime", level=2)
|
||||
items = avg_jct_dict.items()
|
||||
for item in items:
|
||||
key = item[0]
|
||||
print_key = "(%-4s,%4s)" % (ALLOC_POLICY_DICT.get(key[0]), PREEMPT_POLICY_DICT.get(key[1]))
|
||||
print_str = "%s,%.2f,%.2f,%.0f,%d,%.2f" % (print_key, avg_jct_dict[key], wait_time_dict[key], makespan_dict[key], num_jobs_dict[key], runtime_dict[key])
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
print("\nlog_file: %s" % log_file)
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
# usage: python3 run_simulator.py & date
|
||||
|
||||
from simulator import Simulator
|
||||
from utils import print_fn, ALLOC_POLICY_DICT, PREEMPT_POLICY_DICT
|
||||
import os
|
||||
import time
|
||||
import logging
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
DATE = "%02d%02d" % (time.localtime().tm_mon, time.localtime().tm_mday)
|
||||
|
||||
# INPUT TRACE FILE
|
||||
CSV_FILE_PATH = Path(__file__).parent / 'traces/pai/'
|
||||
DESCRIBE_FILE = None
|
||||
CSV_FILE = 'pai_job_no_estimate_100K.csv'
|
||||
|
||||
parser = argparse.ArgumentParser(description='Simulator.')
|
||||
parser.add_argument("-r", "--arrival_rate", help="Arrival Rate", type=int, default=1000)
|
||||
parser.add_argument("-n", "--num_jobs", help="Num of Jobs", type=int, default=1000)
|
||||
parser.add_argument("-g", "--num_gpus", help="Num of GPUs", type=int, default=6500)
|
||||
parser.add_argument("-p", "--repeat", help='Repeat', type=int, default=1)
|
||||
parser.add_argument("-k", "--pack", dest='packing_policy', action='store_true')
|
||||
parser.add_argument("-b", "--balance", dest='packing_policy', action='store_false')
|
||||
parser.set_defaults(packing_policy=False)
|
||||
args = parser.parse_args()
|
||||
NUM_JOBS = args.num_jobs
|
||||
ARRIVAL_RATE = args.arrival_rate
|
||||
NUM_GPUS = args.num_gpus
|
||||
REPEAT = args.repeat
|
||||
SORT_NODE_POLICY = 0 if args.packing_policy is True else 3 # 0: packing, 3: max-min balancing.
|
||||
|
||||
MAX_TIME = int(1e9)
|
||||
|
||||
VERBOSE = 0
|
||||
# VERBOSE = 1
|
||||
# LOG_LEVEL = logging.DEBUG
|
||||
# LOG_LEVEL = logging.INFO
|
||||
LOG_LEVEL = logging.WARNING
|
||||
|
||||
NUM_NODES = 1
|
||||
NUM_CPUS = round(23.22 * NUM_GPUS) # 23.22 * num_gpus 156576/6742
|
||||
HETERO = True # heterogeneous cluster
|
||||
# HETERO = False
|
||||
PATTERN = 0 # Cluster capacity varying pattern
|
||||
|
||||
# GPU_TYPE_MATCHING = 1 # GPU type perfect match
|
||||
GPU_TYPE_MATCHING = 2 # Only V100 cannot compromise
|
||||
# GPU_TYPE_MATCHING = 0
|
||||
|
||||
# EXPORT_JOB_STATS = False
|
||||
EXPORT_JOB_STATS = True
|
||||
# EXPORT_CLUSTER_UTIL = False
|
||||
EXPORT_CLUSTER_UTIL = True
|
||||
|
||||
# RANDOM_SEED = random.randint(0, 100)
|
||||
RANDOM_SEED = 42
|
||||
NUM_SPARE_NODE = 0
|
||||
# SORT_BY_JCT = False
|
||||
SORT_BY_JCT = True
|
||||
|
||||
# Logging in directory
|
||||
LOG_DIR = Path(__file__).parent / 'logs'
|
||||
|
||||
comments = '%dg_%dn_h%d_%dp_%dsn_%dgt-%dar-%dj-%dx-%dr' % (NUM_GPUS, NUM_NODES, HETERO, PATTERN, SORT_NODE_POLICY, GPU_TYPE_MATCHING, ARRIVAL_RATE, NUM_JOBS, REPEAT, RANDOM_SEED)
|
||||
|
||||
log_time = int(time.time() % 100000)
|
||||
if not os.path.exists(LOG_DIR):
|
||||
os.makedirs(LOG_DIR)
|
||||
|
||||
log_file = LOG_DIR / ("%s-%s-%s-%s.log" % (DATE, CSV_FILE, log_time, comments))
|
||||
logging.basicConfig(level=LOG_LEVEL, format="%(message)s", filename=log_file, filemode='a')
|
||||
describe_file = CSV_FILE_PATH / DESCRIBE_FILE if DESCRIBE_FILE is not None else None
|
||||
|
||||
results_dict = {}
|
||||
num_jobs_dict = {}
|
||||
avg_jct_dict = {}
|
||||
makespan_dict = {}
|
||||
wait_time_dict = {}
|
||||
runtime_dict = {}
|
||||
|
||||
print("log_file: %s" % log_file)
|
||||
print_str = "==========\n%d_Jobs_repeated_%d_times\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime" % (NUM_JOBS, REPEAT)
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
for alloc_policy in [8]: # 8FIFO (see utils.py)
|
||||
for preempt_policy in [2]: # 2LGF
|
||||
key = (alloc_policy, preempt_policy)
|
||||
print_key = "(%-4s,%4s)" % (ALLOC_POLICY_DICT.get(key[0]), PREEMPT_POLICY_DICT.get(key[1]))
|
||||
|
||||
# running
|
||||
start_time = time.time()
|
||||
print_fn("\n###### %s ######" % print_key)
|
||||
|
||||
simulator = Simulator(
|
||||
csv_file=CSV_FILE_PATH / CSV_FILE,
|
||||
alloc_policy=alloc_policy,
|
||||
preempt_policy=preempt_policy,
|
||||
sort_node_policy=SORT_NODE_POLICY,
|
||||
num_nodes=NUM_NODES,
|
||||
random_seed=RANDOM_SEED,
|
||||
max_time=MAX_TIME,
|
||||
num_spare_node=NUM_SPARE_NODE,
|
||||
pattern=PATTERN,
|
||||
hetero=HETERO,
|
||||
num_gpus=NUM_GPUS,
|
||||
num_cpus=NUM_CPUS,
|
||||
describe_file=describe_file,
|
||||
log_file=log_file,
|
||||
export_job_stats=EXPORT_JOB_STATS,
|
||||
export_cluster_util=EXPORT_CLUSTER_UTIL,
|
||||
arrival_rate=ARRIVAL_RATE,
|
||||
num_jobs_limit=NUM_JOBS,
|
||||
gpu_type_matching=GPU_TYPE_MATCHING,
|
||||
verbose=VERBOSE)
|
||||
results = simulator.simulator_go(repeat=REPEAT)
|
||||
|
||||
# post processing
|
||||
num_jobs, avg_jct, makespan, wait_time = 0, 0, 0, 0
|
||||
for item in results: # [num_jobs, avg_jct, makespan, [#alloc, alloc_time, #preempt, preempt_time]]
|
||||
num_jobs += item[0]
|
||||
avg_jct += item[1]
|
||||
wait_time += item[2]
|
||||
makespan += item[3]
|
||||
# key = (alloc_policy, preempt_policy)
|
||||
results_dict[key] = results
|
||||
num_jobs_dict[key] = num_jobs
|
||||
avg_jct_dict[key] = avg_jct / REPEAT
|
||||
makespan_dict[key] = makespan / REPEAT
|
||||
wait_time_dict[key] = wait_time / REPEAT
|
||||
runtime_dict[key] = time.time() - start_time
|
||||
|
||||
# print_fn("##### %s runtime: %.4f sec #####\n" % (print_key, runtime_dict[key]))
|
||||
|
||||
print_str = "%s,%.2f,%.2f,%.0f,%d,%.2f" % (print_key, avg_jct_dict[key], wait_time_dict[key], makespan_dict[key], num_jobs_dict[key], runtime_dict[key])
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
if SORT_BY_JCT:
|
||||
print("\n# Sort by JCT")
|
||||
print_fn("\n# Sort by JCT\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime", level=2)
|
||||
items = sorted(avg_jct_dict.items(), key=lambda d: d[1])
|
||||
else:
|
||||
print("\n# Summary")
|
||||
print_fn("\n# Summary\nalloc,preempt,avg_jct,wait_time,makespan,jobs_done,runtime", level=2)
|
||||
items = avg_jct_dict.items()
|
||||
for item in items:
|
||||
key = item[0]
|
||||
print_key = "(%-4s,%4s)" % (ALLOC_POLICY_DICT.get(key[0]), PREEMPT_POLICY_DICT.get(key[1]))
|
||||
print_str = "%s,%.2f,%.2f,%.0f,%d,%.2f" % (print_key, avg_jct_dict[key], wait_time_dict[key], makespan_dict[key], num_jobs_dict[key], runtime_dict[key])
|
||||
print(print_str)
|
||||
print_fn(print_str, level=2)
|
||||
|
||||
print("\nlog_file: %s" % log_file)
|
||||
|
|
@ -1,234 +0,0 @@
|
|||
import utils
|
||||
from utils import print_fn, ALLOC_POLICY_DICT, PREEMPT_POLICY_DICT, _repr_job_concise
|
||||
|
||||
|
||||
class Scheduler:
|
||||
def __init__(self, alloc_policy=0, preempt_policy=0, sort_node_policy=0, cluster=None, gpu_type_matching=0, verbose=0):
|
||||
self.cluster = cluster
|
||||
self.alloc_policy = alloc_policy
|
||||
self.preempt_policy = preempt_policy
|
||||
self.sort_node_policy = sort_node_policy
|
||||
self.node_rotate_counter = 0
|
||||
self.verbose = verbose
|
||||
self.gpu_type_matching = gpu_type_matching
|
||||
# To skip unnecessary self.alloc_job_sort()
|
||||
self.last_time_snapshot = [0, 0, 0, 0] # [idle_gpu, idle_cpu, len(job_list), len(job_to_allocate_cache)]
|
||||
self.cannot_counter = 0
|
||||
|
||||
def alloc_job(self, cluster=None):
|
||||
cluster = cluster if cluster is not None else self.cluster
|
||||
job_list = cluster.job_list # Take cluster.job_list
|
||||
|
||||
# Trying skipping allocation as early as possible
|
||||
if len(job_list) <= 0:
|
||||
return 0
|
||||
ig, ic = cluster.idl_gpus, cluster.idl_cpus
|
||||
this_time_snapshot = [ig, ic, len(job_list), 0] # 0: no job allocated.
|
||||
if self.last_time_snapshot == this_time_snapshot: # exactly the same
|
||||
if self.verbose:
|
||||
print_fn("[{}] Last time snapshot == this time snapshot: {}. Bypass.".format(self.cluster.cur_time, this_time_snapshot))
|
||||
return 0
|
||||
job_min_gpu, job_min_cpu = min(job_list, key=lambda j: j['num_inst'] * j['num_gpu']), min(job_list, key=lambda j: j['num_inst'] * j['num_cpu'])
|
||||
if (ig <= 0 or job_min_gpu['num_inst'] * job_min_gpu['num_gpu'] > ig) and (ic <= 0 or job_min_cpu['num_inst'] * job_min_cpu['num_cpu'] > ic):
|
||||
self.last_time_snapshot = this_time_snapshot
|
||||
return 0
|
||||
|
||||
if self.verbose:
|
||||
print_fn("job_min_gpu, job_min_cpu = {:.1f}, {:.1f}".format(job_min_gpu['num_gpu'], job_min_cpu['num_cpu']))
|
||||
|
||||
job_to_allocate_cache = []
|
||||
# Greedy algorithm or Greedy + load balancing
|
||||
if self.alloc_policy in ALLOC_POLICY_DICT.keys():
|
||||
# Heavy action
|
||||
self.alloc_job_sort(job_list, cluster.job_runn_list)
|
||||
for job_a in job_list:
|
||||
succ_alloc = self.try_allocate_job_to_cluster(job_a, cluster)
|
||||
if succ_alloc == 1:
|
||||
job_to_allocate_cache.append(job_a)
|
||||
elif succ_alloc == -1:
|
||||
break
|
||||
# else, e.g., succ_alloc == 0: pass/continue
|
||||
else:
|
||||
raise KeyError("Uncaptured Allocation Policy Input: %d" % self.alloc_policy)
|
||||
|
||||
this_time_snapshot[-1] = len(job_to_allocate_cache) # num of jobs allocated
|
||||
self.last_time_snapshot = this_time_snapshot
|
||||
for job_a in job_to_allocate_cache:
|
||||
cluster.job_list.remove(job_a)
|
||||
|
||||
def alloc_job_sort(self, job_list, job_runn_list=None):
|
||||
if self.alloc_policy == 0: # short_duration_first
|
||||
job_list.sort(key=lambda e: (e['duration'], e['job_id']))
|
||||
elif self.alloc_policy == 8: # FIFO, remains the original order
|
||||
job_list.sort(key=lambda e: (e['submit_time'], e['job_id']))
|
||||
elif self.alloc_policy in [1, 2, 4]: # SJF with duration estimation
|
||||
est_feature = {1: 'user_dur', 2: 'group_dur', 4: 'group_gpu_dur'}[self.alloc_policy]
|
||||
job_list.sort(key=lambda e: (e[est_feature], e['job_id']))
|
||||
else:
|
||||
raise Exception("Unexpected alloc policy: %d" % self.alloc_policy)
|
||||
|
||||
if self.verbose:
|
||||
for i, j in enumerate(job_list):
|
||||
print_fn("%2d %s" % (i, j))
|
||||
if i > 20:
|
||||
break
|
||||
|
||||
def try_allocate_job_to_cluster(self, job_a, cluster):
|
||||
"""
|
||||
job_a: job to allocate
|
||||
cluster: target cluster
|
||||
|
||||
return:
|
||||
-1: the cluster is full, stop job picking
|
||||
0: the current job cannot be placed, try next
|
||||
1: the current job has been successfully deployed, need record.
|
||||
"""
|
||||
ig, ic = cluster.idl_gpus, cluster.idl_cpus
|
||||
if ig <= 0 and ic <= 0:
|
||||
return -1
|
||||
elif job_a['num_inst'] * job_a['num_gpu'] > ig or job_a['num_inst'] * job_a['num_cpu'] > ic:
|
||||
return 0
|
||||
else: # with in gpu and cpu limits
|
||||
assigned_node_map = {}
|
||||
assigned_inst_num = 0
|
||||
sorted_node_list = self.sorted_node_list(cluster.node_list)
|
||||
for nid, node in enumerate(sorted_node_list):
|
||||
# <Node-job label matching>
|
||||
if self.gpu_type_matching == 1: # GPU type perfect match
|
||||
if job_a['gpu_type'] != 'CPU' and job_a['gpu_type'] != node.gpu_type:
|
||||
continue # cannot on this node
|
||||
elif self.gpu_type_matching == 2: # Only V100 cannot compromise
|
||||
if job_a['gpu_type'] == 'V100' and job_a['gpu_type'] != node.gpu_type:
|
||||
continue # cannot on this node
|
||||
# </Node-job label matching>
|
||||
|
||||
if job_a['num_inst'] == 1:
|
||||
if job_a['num_gpu'] <= node.idl_gpus and job_a['num_cpu'] <= node.idl_cpus:
|
||||
succ_alloc = node.alloc_job(job_a)
|
||||
assert succ_alloc
|
||||
job_a['node'] = node.id
|
||||
print_fn("%sON : N[%d] %s" % (cluster.log_prefix, job_a['node'], job_a))
|
||||
self.display_node_status(cur_node_id=job_a['node'])
|
||||
return 1
|
||||
else: # gang-scheduling: all or nothing
|
||||
node_idle_gpus, node_idle_cpus = node.idl_gpus, node.idl_cpus
|
||||
node_inst_num_gpu, node_inst_num_cpu = job_a['num_inst'], job_a['num_inst'] # init.
|
||||
if job_a['num_gpu'] != 0:
|
||||
node_inst_num_gpu = node_idle_gpus // job_a['num_gpu']
|
||||
if job_a['num_cpu'] != 0:
|
||||
node_inst_num_cpu = node_idle_cpus // job_a['num_cpu']
|
||||
node_inst_num = min(node_inst_num_gpu, node_inst_num_cpu)
|
||||
|
||||
if assigned_inst_num + node_inst_num >= job_a['num_inst']:
|
||||
node_inst_num = job_a['num_inst'] - assigned_inst_num
|
||||
assigned_node_map[nid] = node_inst_num
|
||||
assigned_inst_num += node_inst_num
|
||||
break
|
||||
elif node_inst_num > 0:
|
||||
assigned_node_map[nid] = node_inst_num
|
||||
assigned_inst_num += node_inst_num
|
||||
|
||||
if assigned_inst_num < job_a['num_inst']:
|
||||
print_fn("Cannot allocate all instances (%d/%d) of %s." % (assigned_inst_num, job_a['num_inst'], _repr_job_concise(job_a)))
|
||||
self.cannot_counter += 1
|
||||
if self.cannot_counter % 100000 == 0:
|
||||
print_fn("[%s] %d rejects. len(job_done_list) = %d. Current job: %s." % (cluster.log_prefix, self.cannot_counter, len(self.cluster.job_history.job_done_list), _repr_job_concise(job_a)))
|
||||
return 0 # No successful allocation, for num_inst=1 and >1 cases
|
||||
else:
|
||||
# Successfully Scheduled. Assigning instances to nodes according to the map
|
||||
inst_id = 0
|
||||
for nid, num_inst in assigned_node_map.items():
|
||||
node = sorted_node_list[nid]
|
||||
job_tmp = {'node': -1}
|
||||
for _ in range(num_inst):
|
||||
job_tmp = job_a.copy()
|
||||
job_tmp['inst_id'] = inst_id
|
||||
succ_alloc = node.alloc_job(job_tmp)
|
||||
assert succ_alloc
|
||||
job_tmp['node'] = node.id
|
||||
print_fn("%sON : N[%d] %s Inst[%d]" % (cluster.log_prefix, job_tmp['node'], job_tmp, inst_id))
|
||||
inst_id += 1
|
||||
self.display_node_status(cur_node_id=job_tmp['node'])
|
||||
assert inst_id == job_a['num_inst']
|
||||
return 1
|
||||
|
||||
def sorted_node_list(self, node_list):
|
||||
policy = self.sort_node_policy
|
||||
if policy == 0:
|
||||
node_list.sort(key=lambda n: n.id) # by id
|
||||
elif policy == 1:
|
||||
node_list.sort(key=lambda n: n.idl_gpus) # smallest idle gpus first
|
||||
elif policy == 2:
|
||||
node_list.sort(key=lambda n: -n.idl_gpus) # largest idle gpus first
|
||||
elif policy == 3:
|
||||
node_list.sort(key=lambda n: n.util_rate) # lowest avg. util. first
|
||||
else:
|
||||
node_list.sort(key=lambda n: n.id)
|
||||
return node_list
|
||||
|
||||
def preempt_job(self, cluster=None):
|
||||
cluster = cluster if cluster is not None else self.cluster
|
||||
if all([n.idl_gpus for n in cluster.node_list]) >= 0 and \
|
||||
all([n.idl_cpus for n in cluster.node_list]) >= 0:
|
||||
return 0 # No resource contention, bypass preemption
|
||||
|
||||
preempted_job_list = []
|
||||
if self.preempt_policy in PREEMPT_POLICY_DICT.keys():
|
||||
# Pre node preemption: self.preempt_job_node(node)
|
||||
for node in cluster.node_list:
|
||||
# As long as the resources are sufficient, no proactive preempt for now.
|
||||
if node.idl_gpus < 0 or node.idl_cpus < 0 or len(preempted_job_list) > 0:
|
||||
print_fn("%sPreempt jobs on %s" % (cluster.log_prefix, node))
|
||||
preempted_job_list = self.preempt_job_node(node, preempted_job_list)
|
||||
for job in preempted_job_list:
|
||||
print_fn("%sOFF : %s" % (cluster.log_prefix, job))
|
||||
else:
|
||||
raise NotImplementedError("Preempting job policies not implemented")
|
||||
|
||||
for job in preempted_job_list:
|
||||
cluster.job_list.append(job)
|
||||
# Update Job
|
||||
job['wasted'] += job['progress']
|
||||
job['on_time'] = 0
|
||||
job['progress'] = 0
|
||||
job['node'] = None
|
||||
|
||||
def preempt_job_node(self, node, preempted_job_list):
|
||||
# Svc is updated, but the job is not
|
||||
node.update_idl_gpus()
|
||||
node.update_idl_cpus()
|
||||
|
||||
if self.preempt_policy in PREEMPT_POLICY_DICT.keys():
|
||||
# Sort node.job_runn_list in place
|
||||
self.preempt_job_sort_node(node=node, preempt_policy=self.preempt_policy)
|
||||
|
||||
for job_i in preempted_job_list:
|
||||
for job_j in node.job_runn_list:
|
||||
if job_i['job_id'] == job_j['job_id']: # these instances belong to the same job
|
||||
succ = node.release_job(job_i)
|
||||
assert succ is True
|
||||
preempted_job_list.append(job_i)
|
||||
|
||||
while node.idl_gpus < 0 or node.idl_cpus < 0:
|
||||
job_to_preempt = node.job_runn_list[0]
|
||||
succ = node.release_job(job_to_preempt)
|
||||
assert succ is True
|
||||
preempted_job_list.append(job_to_preempt)
|
||||
|
||||
else:
|
||||
raise KeyError("Uncaptured Preemption Policy Input: %d" % self.preempt_policy)
|
||||
|
||||
return preempted_job_list
|
||||
|
||||
def preempt_job_sort_node(self, node, preempt_policy):
|
||||
if preempt_policy == 1: # small_size_first
|
||||
node.job_runn_list.sort(key=lambda e: (e['size'], e['job_id']))
|
||||
elif preempt_policy == 2: # large_gang_first
|
||||
node.job_runn_list.sort(key=lambda e: (-e['num_gpu'], e['job_id']))
|
||||
else: # preempt_policy==0 or others: short_duration_first
|
||||
node.job_runn_list.sort(key=lambda e: (e['duration'], e['job_id']))
|
||||
|
||||
def display_node_status(self, cur_node_id):
|
||||
if cur_node_id >= 0:
|
||||
cur_node = self.cluster.node_dict[cur_node_id]
|
||||
print_fn(cur_node)
|
||||
|
|
@ -1,236 +0,0 @@
|
|||
# Date: 03.28
|
||||
# Note: 2D: CPU + GPU
|
||||
|
||||
import random
|
||||
import csv
|
||||
import copy
|
||||
import numpy as np
|
||||
from utils import print_fn, _add_job, _repr_job_done, _add_describe, GPU_TYPE_INT_DICT
|
||||
from cluster import Cluster
|
||||
from node import Node
|
||||
from scheduler import Scheduler
|
||||
|
||||
""" Jobs
|
||||
OrderedDict
|
||||
keys: job_id,duration,resource=[num_gpu, num_cpu]
|
||||
duration,size,on_time,wasted,jct,
|
||||
# submit_time,model_name,iterations,interval
|
||||
"""
|
||||
|
||||
|
||||
class Simulator:
|
||||
def __init__(self, csv_file, alloc_policy=0, preempt_policy=0,
|
||||
sort_node_policy=0, oracle=False, random_seed=42,
|
||||
max_time=int(1e10), num_gpus=None, num_cpus=None, num_nodes=4,
|
||||
pattern=1, delta=1, num_spare_node=1,
|
||||
hetero=False, describe_file=None, export_job_stats=False,
|
||||
export_cluster_util=False, log_file=None, arrival_rate=None,
|
||||
arrival_interval=60, arrival_shuffle=False,
|
||||
num_jobs_limit=None, gpu_type_matching=0, verbose=0):
|
||||
self.cluster = None
|
||||
self.scheduler = None
|
||||
self.cur_time = None
|
||||
self.num_jobs = None
|
||||
self.job_list = []
|
||||
self.job_runn_list = []
|
||||
self.exit_flag = None
|
||||
self.max_time = max_time
|
||||
self.num_gpus = num_gpus
|
||||
self.num_cpus = num_cpus
|
||||
self.num_nodes = num_nodes
|
||||
self.csv_file = csv_file
|
||||
self.oracle = oracle # know fluctuation?
|
||||
self.alloc_policy = alloc_policy
|
||||
self.preempt_policy = preempt_policy
|
||||
self.sort_node_policy = sort_node_policy
|
||||
self.describe_dict = _add_describe(describe_file) # describe_file: each users' job distribution in csv format.
|
||||
self.arrival_rate = arrival_rate # job arrival rate
|
||||
self.arrival_interval = arrival_interval
|
||||
self.arrival_shuffle = arrival_shuffle
|
||||
self.num_jobs_limit = num_jobs_limit
|
||||
self.job_origin_list = self.add_job(self.csv_file, self.describe_dict, limit=num_jobs_limit * 10)
|
||||
self.pattern = pattern # which resource varying pattern?
|
||||
self.delta = delta # time granularity, minimum: 1 second.
|
||||
self.num_spare_node = num_spare_node
|
||||
self.hetero = hetero
|
||||
self.gpu_type_matching = gpu_type_matching
|
||||
self.export_job_stats = export_job_stats
|
||||
self.export_cluster_util = export_cluster_util
|
||||
self.log_file = log_file # just pass the path
|
||||
self.verbose = verbose
|
||||
random.seed(random_seed)
|
||||
|
||||
@staticmethod
|
||||
def add_job(csv_file, describe_dict, limit=None):
|
||||
"""
|
||||
limit: To avoid reading too many jobs when the sampled number << total number of jobs in trace file.
|
||||
"""
|
||||
job_list = []
|
||||
with open(csv_file, 'r') as fd:
|
||||
reader = csv.DictReader(fd, delimiter=',')
|
||||
keys = reader.fieldnames
|
||||
for i, row in enumerate(reader):
|
||||
_add_job(job_list, row, describe_dict)
|
||||
if limit is not None and i >= limit:
|
||||
break
|
||||
return job_list
|
||||
|
||||
@staticmethod
|
||||
def set_job_list_arrival_time(job_list, arrival_rate=None, interval=60, shuffle_order=False):
|
||||
"""
|
||||
job_list: jobs to execute in this run
|
||||
arrival_rate: num of jobs to arrive at each time interval (-1 or None means no changes)
|
||||
interval: time interval (default: 60)
|
||||
shuffle_order: bool, whether each user's inherent job order are shuffled (default: False)
|
||||
"""
|
||||
if arrival_rate is None or arrival_rate < 0:
|
||||
return 0 # respect the original submit time
|
||||
if shuffle_order is True:
|
||||
np.random.shuffle(job_list)
|
||||
else:
|
||||
job_list.sort(key=lambda e: (e.get('submit_time', float('inf')), e['job_id']))
|
||||
|
||||
arrival_counter = 0
|
||||
for job in job_list:
|
||||
arrival_time = (arrival_counter // arrival_rate) * interval
|
||||
job['submit_time'] = arrival_time
|
||||
arrival_counter += 1
|
||||
|
||||
def init_node_list(self):
|
||||
return [Node(id=i) for i in range(self.num_nodes)]
|
||||
|
||||
def init_node_list_hetero(self):
|
||||
node_list = []
|
||||
node_id = 0
|
||||
for _ in range(16): # low-ended machines first
|
||||
node_list.append(Node(node_id, 0, 96, gpu_type='CPU'))
|
||||
node_id += 1
|
||||
for _ in range(56): # low-ended machines first
|
||||
node_list.append(Node(node_id, 8, 96, gpu_type='MISC'))
|
||||
node_id += 1
|
||||
for _ in range(100): # low-ended machines first
|
||||
node_list.append(Node(node_id, 2, 96, gpu_type='T4'))
|
||||
node_id += 1
|
||||
for _ in range(160): # low-ended machines first
|
||||
node_list.append(Node(node_id, 2, 64, gpu_type='P100'))
|
||||
node_id += 1
|
||||
for _ in range(48):
|
||||
node_list.append(Node(node_id, 8, 96, gpu_type='V100'))
|
||||
node_id += 1
|
||||
return node_list
|
||||
|
||||
def init_go(self, num_jobs=None):
|
||||
self.cur_time = 0
|
||||
self.job_list = copy.deepcopy(self.job_origin_list) # copy each obj in the list
|
||||
num_jobs = num_jobs if num_jobs is not None else self.num_jobs_limit
|
||||
if (num_jobs is not None) and num_jobs <= len(self.job_list):
|
||||
random.shuffle(self.job_list)
|
||||
self.job_list = self.job_list[:num_jobs]
|
||||
self.set_job_list_arrival_time(self.job_list, self.arrival_rate, self.arrival_interval, self.arrival_shuffle)
|
||||
print_fn("----------------------------- RANDOM: %d" % random.randint(1000, 9999))
|
||||
print_fn("%d Job loaded" % len(self.job_list))
|
||||
|
||||
# Init Cluster resources
|
||||
if self.hetero:
|
||||
node_list = self.init_node_list_hetero()
|
||||
elif self.num_nodes == 1 and self.num_gpus is not None: # i.e., one big node formulation
|
||||
node_list = [Node(id=1, num_gpus=self.num_gpus, num_cpus=self.num_cpus)]
|
||||
else:
|
||||
node_list = self.init_node_list()
|
||||
self.cluster = Cluster(node_list=node_list, job_list=self.job_list, random_seed=random.randint(1000, 9999),
|
||||
num_spare_node=self.num_spare_node, pattern=self.pattern,
|
||||
export_cluster_util=self.export_cluster_util)
|
||||
|
||||
self.scheduler = Scheduler(cluster=self.cluster, alloc_policy=self.alloc_policy,
|
||||
preempt_policy=self.preempt_policy, sort_node_policy=self.sort_node_policy,
|
||||
verbose=self.verbose, gpu_type_matching=self.gpu_type_matching)
|
||||
self.num_jobs = len(self.job_list)
|
||||
self.exit_flag = 0
|
||||
|
||||
print_fn("Spared nodes: %s" % self.cluster.spare_node_id)
|
||||
|
||||
def exp_summary(self, id=None):
|
||||
job_history = self.cluster.job_history
|
||||
num_jobs_done = job_history.num_jobs_done
|
||||
jct_summary = job_history.jct_summary
|
||||
wait_time_summary = job_history.wait_time_summary
|
||||
job_done_list = job_history.job_done_list
|
||||
wasted_summary = job_history.wasted_summary
|
||||
assert num_jobs_done == len(job_done_list)
|
||||
|
||||
print_fn("Wasted progress in summary: %s" % wasted_summary)
|
||||
if num_jobs_done == self.num_jobs:
|
||||
print_fn("All Done (makespan) at %s" % self.cur_time)
|
||||
else:
|
||||
print_fn("%d of %d jobs Done (makespan) at %s" % (num_jobs_done, self.num_jobs, self.cur_time))
|
||||
print_fn("%d jobs' average JCT: %.4f, average wait time: %.4f" % (num_jobs_done, jct_summary / num_jobs_done, wait_time_summary / num_jobs_done))
|
||||
# Print job done breakdown
|
||||
job_done_list.sort(key=lambda e: e['job_id'])
|
||||
|
||||
if self.export_job_stats is True:
|
||||
job_stats = np.zeros(shape=(6, len(job_done_list)), dtype=int)
|
||||
for i, job in enumerate(job_done_list):
|
||||
print_fn("%s || %s" % (_repr_job_done(job), job))
|
||||
job_stats[0][i] = job['submit_time']
|
||||
job_stats[1][i] = job['duration']
|
||||
job_stats[2][i] = job['jct']
|
||||
job_stats[3][i] = GPU_TYPE_INT_DICT.get(job.get('gpu_type', 'N/A'), -1)
|
||||
job_stats[4][i] = job.get('num_inst', 1)
|
||||
job_stats[5][i] = job['job_id']
|
||||
job_stats_name = "%s.a%s-p%s-i%s-job_stats.npy" % (
|
||||
self.log_file.name, self.alloc_policy, self.preempt_policy, id)
|
||||
job_stats_file = self.log_file.parent / job_stats_name
|
||||
np.save(job_stats_file, job_stats)
|
||||
else:
|
||||
for job in job_done_list:
|
||||
print_fn("%s || %s" % (_repr_job_done(job), job))
|
||||
print_fn("")
|
||||
|
||||
if self.export_cluster_util is True:
|
||||
cluster_util = np.asarray([
|
||||
self.cluster.cluster_time,
|
||||
self.cluster.cluster_cpu,
|
||||
self.cluster.cluster_gpu
|
||||
])
|
||||
cluster_util_name = "%s.a%s-p%s-i%s-cluster_util.npy" % (
|
||||
self.log_file.name, self.alloc_policy, self.preempt_policy, id)
|
||||
cluster_util_file = self.log_file.parent / cluster_util_name
|
||||
np.save(cluster_util_file, cluster_util)
|
||||
return num_jobs_done, jct_summary, wait_time_summary
|
||||
|
||||
def simulator_go(self, repeat=1, num_jobs=None):
|
||||
"""
|
||||
:return: [[num_jobs, avg_jct, wait_time, makespan], [], ... ]
|
||||
"""
|
||||
result = []
|
||||
for repeat_id in range(repeat):
|
||||
self.init_go(num_jobs=num_jobs)
|
||||
|
||||
while not self.exit_flag:
|
||||
self.tic(self.delta)
|
||||
|
||||
num_jobs_done, jct_summary, wait_time_summary = self.exp_summary(repeat_id)
|
||||
result.append((num_jobs_done, jct_summary / num_jobs_done, wait_time_summary / num_jobs_done, self.cur_time))
|
||||
return result
|
||||
|
||||
def tic(self, delta=1):
|
||||
if self.cur_time < self.max_time:
|
||||
self.cluster.tic_svc(self.cur_time)
|
||||
|
||||
# Preempt job
|
||||
self.scheduler.preempt_job(self.cluster)
|
||||
|
||||
# Allocate job
|
||||
self.scheduler.alloc_job(self.cluster)
|
||||
|
||||
# Jobs tic and global cur_time += delta
|
||||
tic_return_value = self.cluster.tic_job(delta)
|
||||
if tic_return_value >= 0:
|
||||
self.cur_time = tic_return_value
|
||||
else:
|
||||
self.cur_time = self.cur_time + delta
|
||||
self.exit_flag = 1
|
||||
else:
|
||||
print_fn("TIMEOUT {} with jobs {}".format(self.cur_time, self.cluster.job_list))
|
||||
self.exit_flag = 1
|
||||
raise TimeoutError("TIMEOUT {} with jobs {}".format(self.cur_time, self.cluster.job_list))
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -1,235 +0,0 @@
|
|||
import logging
|
||||
import csv
|
||||
import numpy as np
|
||||
from matplotlib import pyplot as plt
|
||||
|
||||
ALLOC_POLICY_DICT = {
|
||||
0: 'SJF', # 'short job first', SJF
|
||||
1: 'SJU', # SJF with estimator using USER feature
|
||||
2: 'SJG', # SJF with estimator using GROUP, USER feature
|
||||
4: 'SJGG', # SJF with estimator using GROUP, USER, GPU feature
|
||||
8: 'FIFO', # FIFO, the default
|
||||
}
|
||||
|
||||
PREEMPT_POLICY_DICT = {
|
||||
0: 'SDF', # 'smallest_duration_first'
|
||||
1: 'SSF', # 'smallest_size_first
|
||||
2: 'LGF', # 'large_gpu_first', # LGF, size:num_gpu
|
||||
}
|
||||
|
||||
GPU_TYPE_INT_DICT = {
|
||||
"CPU": 0,
|
||||
"MISC": 1,
|
||||
"T4": 2,
|
||||
"P100": 3,
|
||||
"V100": 4
|
||||
}
|
||||
|
||||
|
||||
def print_fn(log, level=1):
|
||||
LOG_LEVEL_DEBUG = 0
|
||||
LOG_LEVEL_INFO = 1
|
||||
LOG_LEVEL_WARNING = 2
|
||||
LOG_LEVEL_ERROR = 3
|
||||
if level == LOG_LEVEL_DEBUG:
|
||||
logging.debug(log)
|
||||
elif level == LOG_LEVEL_INFO:
|
||||
logging.info(log)
|
||||
elif level == LOG_LEVEL_WARNING:
|
||||
logging.warning(log)
|
||||
elif level == LOG_LEVEL_ERROR:
|
||||
logging.error(log)
|
||||
exit()
|
||||
|
||||
|
||||
def _repr_job_concise(job_dict):
|
||||
return "J %s([G %s,C %s]-D %s)" % (job_dict['job_id'], job_dict['num_gpu'], job_dict['num_cpu'], job_dict['duration'])
|
||||
|
||||
|
||||
def _repr_job_preempt(job_dict):
|
||||
return "J %s-[G %s,C %s]-O:%3s/D:%3s" % (job_dict['job_id'], job_dict['num_gpu'], job_dict['num_cpu'], job_dict['on_time'], job_dict['duration'])
|
||||
|
||||
|
||||
def _repr_job_done(job_dict):
|
||||
job_repr_concise = "J %s([G %s,C %s]-D %s-N %s)" % (job_dict['job_id'], job_dict['num_gpu'], job_dict['num_cpu'], job_dict['duration'], job_dict['node'])
|
||||
return "%25s: %4s ---> %4s" % (job_repr_concise, job_dict['jct'] - job_dict['duration'], job_dict['jct'])
|
||||
|
||||
|
||||
def _add_describe(describe_file):
|
||||
if describe_file is None:
|
||||
return None
|
||||
describe_dict = {}
|
||||
with open(describe_file, 'r') as fd:
|
||||
reader = csv.DictReader(fd, delimiter=',')
|
||||
for row in reader:
|
||||
for k, v in row.items():
|
||||
if k=='count':
|
||||
row[k] = int(v)
|
||||
elif k in ['mean', 'std', 'min', '25%', '50%', '75%', 'max']:
|
||||
if v == '':
|
||||
v = 0
|
||||
row[k] = float(v)
|
||||
describe_dict[row['user']] = row
|
||||
return describe_dict # dd['ae8ed1']['50%']==38.4
|
||||
|
||||
|
||||
def _add_job(job_list, job_dict, describe_dict=None):
|
||||
# Add job (job_dict) into job_list
|
||||
for key, value in job_dict.items():
|
||||
if value is not None and value.isdigit() and key != 'user':
|
||||
if type(value) == str:
|
||||
job_dict[key] = round(float(value))
|
||||
else: # duration becomes an int
|
||||
job_dict[key] = round(value)
|
||||
elif key in ['wait_time','user_dur','user_gpu_dur','group_dur','group_gpu_dur']:
|
||||
try:
|
||||
job_dict[key] = float(value)
|
||||
except:
|
||||
pass
|
||||
|
||||
keys = ['num_cpu', 'num_gpu', 'submit_time', 'num_inst']
|
||||
for key in keys:
|
||||
if key not in job_dict or job_dict[key] == '':
|
||||
if key in ['num_cpu', 'num_gpu']:
|
||||
job_dict[key] = 0
|
||||
else: # key in ['submit_time', 'num_inst']
|
||||
job_dict[key] = 1
|
||||
else:
|
||||
if key in ['num_cpu', 'num_gpu']: # in %
|
||||
job_dict[key] = round(100 * float(job_dict[key]))
|
||||
else:
|
||||
job_dict[key] = round(float(job_dict[key]))
|
||||
|
||||
# Add entries to be used in scheduling
|
||||
job_dict['duration'] = int(float(job_dict['duration']))
|
||||
if job_dict['duration'] <= 0:
|
||||
job_dict['duration'] = 1 # fix duration == 0 problem.
|
||||
job_dict['size'] = int((job_dict['num_gpu'] + job_dict['num_cpu']) * job_dict['duration']) # (gpu + cpu) x duration
|
||||
job_dict['on_time'] = 0
|
||||
job_dict['wasted'] = 0
|
||||
job_dict['jct'] = -1
|
||||
job_dict['resource'] = [job_dict['num_gpu'], job_dict['num_cpu']] # list of resources
|
||||
job_dict['node'] = None
|
||||
|
||||
# Add duration estimation
|
||||
if describe_dict is not None:
|
||||
jd_user = describe_dict.get(job_dict['user'])
|
||||
if jd_user is not None:
|
||||
job_dict['dur_avg'] = float(jd_user['mean']) # expectation
|
||||
job_dict['dur_std'] = float(jd_user['std']) # standard deviation
|
||||
job_dict['dur_med'] = float(jd_user['50%']) # median
|
||||
job_dict['dur_trim_mean'] = float(jd_user['trim_mean']) # discard 10% top and 10% tail when calc. mean
|
||||
|
||||
# Remove original unused entries
|
||||
for drop_col in ['fuxi_job_name','fuxi_task_name','inst_id','running_cluster','model_name','iterations','interval','vc','jobid','status']:
|
||||
if drop_col in job_dict: job_dict.pop(drop_col)
|
||||
|
||||
job_list.append(job_dict)
|
||||
|
||||
|
||||
def add_user_round_robin_id(job_list):
|
||||
# Add a new sorting metrics, user_rrid, to enforce scheduler picking jobs from multiple users
|
||||
# when all users' primary metrics are the same (e.g., 0).
|
||||
user_rrid_dict = {} # a new dict each time
|
||||
for job in job_list:
|
||||
user = job['user']
|
||||
rrid = user_rrid_dict.get(user, None)
|
||||
if rrid is None:
|
||||
rrid = 0
|
||||
user_rrid_dict[user] = 1
|
||||
else:
|
||||
user_rrid_dict[user] += 1
|
||||
job['user_rrid'] = rrid
|
||||
|
||||
|
||||
def large_job_pruning(job_list, gpu_limit, cpu_limit):
|
||||
if job_list is None:
|
||||
return []
|
||||
for job in job_list:
|
||||
if 'num_gpu' in job and job['num_gpu'] > gpu_limit:
|
||||
gpu_was = job['num_gpu']
|
||||
job['num_gpu'] = gpu_limit
|
||||
print_fn("{:s}: GPU {:d} ==> {:d}".format(_repr_job_concise(job), gpu_was, gpu_limit))
|
||||
if 'num_cpu' in job and job['num_cpu'] > cpu_limit:
|
||||
cpu_was = job['num_cpu']
|
||||
job['num_cpu'] = cpu_limit
|
||||
print_fn("{:s}: CPU {:d} ==> {:d}".format(_repr_job_concise(job), cpu_was, cpu_limit))
|
||||
return job_list
|
||||
|
||||
|
||||
def plot_cluster_util(npyfile, to_date=False):
|
||||
cluster_util = np.load(npyfile)
|
||||
cluster_time, cluster_cpu, cluster_gpu = cluster_util[0], cluster_util[1], cluster_util[2]
|
||||
|
||||
plt.clf()
|
||||
plt.plot(cluster_time, cluster_cpu / 10, label='10CPU')
|
||||
plt.plot(cluster_time, cluster_gpu, label='GPU')
|
||||
plt.legend()
|
||||
try:
|
||||
plt.savefig(str(npyfile).split('.npy')[0]+".png")
|
||||
except:
|
||||
plt.savefig("cluster_util")
|
||||
|
||||
|
||||
def plot_job_stats(npyfile, to_date=False):
|
||||
plt.figure(figsize=(16, 6), dpi=120)
|
||||
job_stats = np.load(npyfile)
|
||||
job_submit_time, job_duration, job_jct, job_gpu_type, job_num_inst, job_id = job_stats[0], job_stats[1], job_stats[2], job_stats[3], job_stats[4], job_stats[5]
|
||||
job_queue_delay = job_jct - job_duration
|
||||
|
||||
plt.clf()
|
||||
plt.plot(job_submit_time, job_queue_delay, color='orange', label='queue_delay')
|
||||
plt.plot(job_submit_time, job_duration, color='black', alpha=0.3, label='duration')
|
||||
plt.legend()
|
||||
try:
|
||||
plt.savefig(str(npyfile).split('.npy')[0]+".png")
|
||||
except:
|
||||
plt.savefig("job_stats")
|
||||
|
||||
|
||||
def plot_multi_job_stats(npyfiles, to_date=False):
|
||||
plt.clf()
|
||||
plt.figure(figsize=(12, 6), dpi=120)
|
||||
|
||||
for npyfile in npyfiles:
|
||||
job_stats = np.load(npyfile)
|
||||
job_submit_time, job_duration, job_jct, job_gpu_type, job_num_inst, job_id = job_stats[0], job_stats[1], job_stats[2], job_stats[3], job_stats[4], job_stats[5]
|
||||
job_queue_delay = job_jct - job_duration
|
||||
|
||||
try:
|
||||
label=ALLOC_POLICY_DICT[int(str(npyfile).split('.log.a')[1].split('-p')[0])]
|
||||
except KeyError:
|
||||
label = str(npyfile).split('.log.')[1].split('-job_stats.npy')[0]
|
||||
plt.plot(job_submit_time, job_queue_delay, alpha=0.5, label=label+'-queue_delay')
|
||||
plt.plot(job_submit_time, job_duration, color='grey', alpha=0.3, label='job duration')
|
||||
plt.legend(loc='upper left')
|
||||
plt.title("Arrival jobs' duration and queueing delay")
|
||||
plt.xlabel("Submitted Time")
|
||||
plt.ylabel("Run/Wait Time")
|
||||
try:
|
||||
plt.savefig(str(npyfile).split('.log.')[0]+"-job_stats.png")
|
||||
except:
|
||||
plt.savefig("job_stats")
|
||||
|
||||
|
||||
def plot_multi_cluster_util(npyfiles, to_date=False):
|
||||
plt.clf()
|
||||
plt.figure(figsize=(12, 6), dpi=120)
|
||||
|
||||
for npyfile in npyfiles:
|
||||
cluster_util = np.load(npyfile)
|
||||
cluster_time, cluster_cpu, cluster_gpu = cluster_util[0], cluster_util[1], cluster_util[2]
|
||||
|
||||
try:
|
||||
label=ALLOC_POLICY_DICT[int(str(npyfile).split('.log.a')[1].split('-p')[0])]
|
||||
except KeyError:
|
||||
label = str(npyfile).split('.log.')[1].split('-cluster_util.npy')[0]
|
||||
plt.plot(cluster_time, cluster_gpu, alpha=0.5, label=label+'-GPU')
|
||||
plt.legend(loc='upper left')
|
||||
plt.title("Cluster Utilization")
|
||||
plt.xlabel("Time")
|
||||
plt.ylabel("Resource")
|
||||
try:
|
||||
plt.savefig(str(npyfile).split('.log.')[0]+"-cluster_util.png")
|
||||
except:
|
||||
plt.savefig("cluster_util")
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
# ignore generated yaml directories
|
||||
openb_pod_list*/
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
# Traces for [Kubernetes Scheduler Simulator](https://github.com/hkust-adsl/kubernetes-scheduler-simulator)
|
||||
|
||||
## ℹ️ Basics
|
||||
|
||||
This repo contains trace data for the [Kubernetes Scheduler Simulator](https://github.com/hkust-adsl/kubernetes-scheduler-simulator), which evaluates different scheduling policies in GPU-sharing clusters.
|
||||
It includes the Fragmentation Gradient Descent (FGD) policy proposed in the [USENIX ATC 2023](https://www.usenix.org/conference/atc23) paper "[Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent](https://www.usenix.org/conference/atc23/presentation/weng)", along with other baseline policies (e.g., Best-fit, Dot-product, GPU Packing, GPU Clustering, Random-fit).
|
||||
|
||||
## 🗄️ Traces
|
||||
|
||||
Key data is in the csv folder, while yaml files can be generated from this data.
|
||||
|
||||
### [openb_node_list_all_node.csv](./csv/openb_node_list_all_node.csv)
|
||||
|
||||
This file contains 1523 nodes of a heterogeneous GPU cluster in production, listing their CPU, main memory, GPU specifications and GPU types.
|
||||
|
||||
[openb_node_list_gpu_node.csv](./csv/openb_node_list_gpu_node.csv) is a subset excluding non-GPU nodes. [openb_node_list_gpu_node.yaml](./node_yaml/openb_node_list_gpu_node.yaml) contains the same data in YAML format.
|
||||
|
||||
Here's a sample output:
|
||||
|
||||
| | sn | cpu_milli | memory_mib | gpu | model |
|
||||
|---:|:----------------|------------:|-------------:|------:|:--------|
|
||||
| 0 | openb-node-0227 | 32000 | 262144 | 0 | nan |
|
||||
| 1 | openb-node-0228 | 128000 | 786432 | 8 | G3 |
|
||||
| 2 | openb-node-0229 | 96000 | 786432 | 8 | V100M32 |
|
||||
|
||||
- `cpu_milli`: Number of CPUs (in milli)
|
||||
- `memory_mib`: Main memory (in MiB)
|
||||
- `gpu`: Number of GPUs
|
||||
- `model`: GPU type. G1, G2, G3 are undisclosed internal GPU codes.
|
||||
|
||||
### [openb_pod_list_default.csv](./csv/openb_pod_list_default.csv)
|
||||
|
||||
This file contains 8152 tasks submitted to the GPU cluster, listing their resource specifications, QoS, phase and creation/deletion/scheduled times.
|
||||
|
||||
The other openb_pod_list_*.csv files (excluding the gpuspec ones) are sampled from the default one, emphasizing certain types of workloads (e.g., CPU-only tasks, GPU-sharing tasks, multi-GPU tasks).
|
||||
|
||||
Trace files with `gpuspec` augment tasks with GPU type requirements. About 33% of GPU tasks in our production cluster have GPU type constraints (see [openb_pod_list_gpuspec33.csv](./csv/openb_pod_list_gpuspec33.csv) and Sec. 6.5 in the "[Beware of Fragmentation](https://www.usenix.org/conference/atc23/presentation/weng)" paper).
|
||||
|
||||
Here's a sample output:
|
||||
|
||||
| | name | cpu_milli | memory_mib | num_gpu | gpu_milli | gpu_spec | qos | pod_phase | creation_time | deletion_time | scheduled_time |
|
||||
|---:|:---------------|------------:|-------------:|----------:|------------:|:----------------|:----------|:------------|----------------:|----------------:|-----------------:|
|
||||
| 0 | openb-pod-0017 | 88000 | 327680 | 8 | 1000 | nan | Burstable | Succeeded | 9437497 | 10769854 | 9437497 |
|
||||
| 1 | openb-pod-0022 | 4000 | 15258 | 1 | 220 | nan | BE | Running | 9679175 | 9973826 | 9679175 |
|
||||
| 2 | openb-pod-0035 | 16000 | 32768 | 1 | 1000 | V100M16\|V100M32 | LS | Running | 9967058 | 9968575 | 9967063 |
|
||||
|
||||
- `cpu_milli`: Number of CPUs requested (in milli)
|
||||
- `memory_mib`: Main memory requested (in MiB)
|
||||
- `num_gpu`: Number of GPUs requested (integers from 0 to 8)
|
||||
- `gpu_milli`: Detailed GPU requested for GPU-sharing workloads (i.e., `num_gpu==1`) (in milli).
|
||||
- `gpu_spec`: Required GPU types, For example, `nan` means no GPU type constraints while `V100M16|V100M32` means the task can run on [NVIDIA V100](https://www.nvidia.com/en-us/data-center/v100/) with either 16GB VRAM or 32GB VRAM.
|
||||
- `qos`: [Quality of Service](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) (e.g., Burstable, Best Effort (BE), Latency Sensitive (LS))
|
||||
- `pod_phrase`: Succeeded, Running, Pending, Failed
|
||||
- `creation_time`: Timestamp of creation (in seconds)
|
||||
- `deletion_time`: Timestamp of deletion (in seconds)
|
||||
- `scheduled_time`: Timestamp of being scheduled (in seconds)
|
||||
|
||||
## 🛠 Usage
|
||||
|
||||
Generate the YAML files needed for the simulation experiment based on the original CSV files.
|
||||
|
||||
```bash
|
||||
$ bash prepare_input.sh
|
||||
```
|
||||
|
||||
For cluster scheduling simulation on Kubernetes, refer to the Kubernetes Scheduler Simulator repo: https://github.com/hkust-adsl/kubernetes-scheduler-simulator.
|
||||
|
||||
## 📝 Paper
|
||||
|
||||
Please cite our paper if it is helpful to your research.
|
||||
|
||||
```
|
||||
@inproceedings{FGD2023,
|
||||
title = {Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent},
|
||||
author = {Qizhen Weng and Lingyun Yang and Yinghao Yu and Wei Wang and Xiaochuan Tang and Guodong Yang and Liping Zhang},
|
||||
booktitle = {2023 {USENIX} Annual Technical Conference},
|
||||
year = {2023},
|
||||
series = {{USENIX} {ATC} '23},
|
||||
url = {https://www.usenix.org/conference/atc23/presentation/weng},
|
||||
publisher = {{USENIX} Association}
|
||||
}
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -1,159 +0,0 @@
|
|||
import sys
|
||||
import yaml
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
|
||||
USAGE_PROMPT="""Usage:
|
||||
python3 pod_csv_to_yaml.py data/csv/openb_pod_list_gpuspec10.csv
|
||||
"""
|
||||
OUTPUT_DIR_DEFAULT="data/new_output"
|
||||
|
||||
MILLI = 1000
|
||||
DATA_CREATION_TIME = "creation_time"
|
||||
DATA_DELETION_TIME = "deletion_time"
|
||||
|
||||
ResourceName = "alibabacloud.com/gpu-milli" # GPU milli, i.e., 1000 == 1 GPU, for pod only, node is 1000 by default
|
||||
CountName = "alibabacloud.com/gpu-count" # GPU number request (or allocatable), for pod and node
|
||||
DeviceIndex = "alibabacloud.com/gpu-index" # Exists when the pod are assigned/predefined to a GPU device
|
||||
ModelName = "alibabacloud.com/gpu-card-model" # GPU card model, for pod and node
|
||||
AssumeTime = "alibabacloud.com/assume-time" # To retrieve the scheduling latency
|
||||
CreationTime = "alibabacloud.com/creation-time" # creation timestamp
|
||||
DeletionTime = "alibabacloud.com/deletion-time" # deletion timestamp
|
||||
PodNsNameSep = "/"
|
||||
DevIdSep = "-"
|
||||
|
||||
def generate_pod_yaml(workload_name='paib-pod-10',
|
||||
workload_namespace='paib-gpu',
|
||||
container_name='main',
|
||||
container_image='tensorflow:latest',
|
||||
container_requests={'cpu': '6000m'},
|
||||
container_limits={'cpu': '6000m'},
|
||||
node_selector_node_ip="",
|
||||
annotations={},
|
||||
labels={}):
|
||||
pod_template = """
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: single-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: php-redis
|
||||
image: gcr.io/google-samples/gb-frontend:v4
|
||||
imagePullPolicy: Always
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
limits:
|
||||
cpu: 100m
|
||||
restartPolicy: "OnFailure"
|
||||
dnsPolicy: "Default"
|
||||
"""
|
||||
workload_yaml = yaml.safe_load(pod_template)
|
||||
workload_yaml['metadata']['name'] = workload_name
|
||||
workload_yaml['metadata']['namespace'] = workload_namespace
|
||||
workload_yaml['spec']['containers'][0]['name'] = container_name
|
||||
workload_yaml['spec']['containers'][0]['image'] = container_image
|
||||
workload_yaml['spec']['containers'][0]['resources']['requests'] = container_requests
|
||||
workload_yaml['spec']['containers'][0]['resources']['limits'] = container_limits
|
||||
|
||||
if len(node_selector_node_ip) > 0:
|
||||
if 'nodeSelector' not in workload_yaml['spec']:
|
||||
workload_yaml['spec']['nodeSelector'] = {}
|
||||
workload_yaml['spec']['nodeSelector']['node-ip'] = node_selector_node_ip
|
||||
elif 'nodeSelector' in workload_yaml['spec']:
|
||||
if 'node-ip' in workload_yaml["spec"]["nodeSelector"]:
|
||||
del workload_yaml['spec']['nodeSelector']['node-ip']
|
||||
|
||||
for k, v in annotations.items():
|
||||
if 'annotations' not in workload_yaml['metadata']:
|
||||
workload_yaml['metadata']['annotations'] = {}
|
||||
if v is not None:
|
||||
workload_yaml['metadata']['annotations'][k] = v # e.g., {"alibabacloud.com/gpu-index":"2-3-4"}
|
||||
for k, v in labels.items():
|
||||
workload_yaml['metadata'][k] = v
|
||||
|
||||
return workload_yaml
|
||||
|
||||
|
||||
def output_pod(dfp, outfile='pod.yaml', node_select=False):
|
||||
num_pod = len(dfp)
|
||||
for index, row in dfp.iterrows():
|
||||
if 'name' in row:
|
||||
workload_name = row['name']
|
||||
elif 'job_id' in row:
|
||||
workload_name = f"job-{row['job_id']:04}" # float is not allowed
|
||||
else:
|
||||
exit("neither name nor job_id in row")
|
||||
|
||||
container_requests = {}
|
||||
if 'cpu_milli' in row:
|
||||
container_requests['cpu'] = "%dm" % (row['cpu_milli'])
|
||||
elif 'cpu' in row:
|
||||
container_requests['cpu'] = "%dm" % (row['cpu'] * MILLI)
|
||||
elif 'num_cpu' in row:
|
||||
container_requests['num_cpu'] = "%dm" % (row['num_cpu'] * MILLI)
|
||||
else:
|
||||
exit("neither cpu_milli nor cpu in row")
|
||||
if 'memory_mib' in row:
|
||||
container_requests['memory'] = "%dMi" % row['memory_mib']
|
||||
container_limits = container_requests.copy()
|
||||
|
||||
host_node_ip = row['ip'] if node_select else ""
|
||||
|
||||
annotations = {}
|
||||
if int(row['num_gpu']) != 0:
|
||||
if node_select:
|
||||
annotations[DeviceIndex] = row['gpu_index'] if type(row['gpu_index']) == str else ""
|
||||
if 'gpu_milli' not in row:
|
||||
annotations[ResourceName] = 1000
|
||||
else:
|
||||
annotations[ResourceName] = "%d" % (int(row['gpu_milli'])) if 0 < row['gpu_milli'] <= 1000 else "1000" if row['gpu_milli'] > 1000 else "0"
|
||||
annotations[CountName] = "%d" % (int(row['num_gpu']))
|
||||
|
||||
if 'gpu_spec' in row:
|
||||
gpu_req_val = [x for x in row['gpu_spec'].split('|') if len(x) > 0]
|
||||
gpu_req_out = "|".join(x for x in gpu_req_val)
|
||||
if len(gpu_req_out) > 0:
|
||||
annotations[ModelName] = gpu_req_out
|
||||
# annotations[CreationTime] = "%s" % row[DATA_CREATION_TIME] if DATA_CREATION_TIME in row else None
|
||||
# annotations[DeletionTime] = "%s" % row[DATA_DELETION_TIME] if DATA_DELETION_TIME in row else None
|
||||
|
||||
pod_yaml = generate_pod_yaml(workload_name=workload_name, container_requests=container_requests,
|
||||
container_limits=container_limits, node_selector_node_ip=host_node_ip,
|
||||
annotations=annotations)
|
||||
|
||||
if index == 0:
|
||||
with open(outfile, 'w') as file:
|
||||
yaml.dump(pod_yaml, file)
|
||||
else:
|
||||
with open(outfile, 'a') as file:
|
||||
file.writelines(['\n---\n\n'])
|
||||
yaml.dump(pod_yaml, file)
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 2:
|
||||
exit(USAGE_PROMPT)
|
||||
pod_csv_file = Path(sys.argv[1])
|
||||
if not pod_csv_file.exists():
|
||||
exit(f"CSV File: {pod_csv_file} does not exist")
|
||||
|
||||
dfp = pd.read_csv(pod_csv_file, dtype={'gpu_index': str})
|
||||
if 'gpu_spec' in dfp:
|
||||
dfp.gpu_spec = dfp.gpu_spec.fillna('')
|
||||
|
||||
output_dir = pod_csv_file.stem # .csv to ""
|
||||
if len(output_dir) <= 0:
|
||||
output_dir_path = Path(OUTPUT_DIR_DEFAULT)
|
||||
else:
|
||||
output_dir_path = Path(output_dir)
|
||||
output_dir_path.mkdir(exist_ok=True)
|
||||
|
||||
pod_yaml_file = output_dir_path / (pod_csv_file.stem + '.yaml') # .csv to .yaml
|
||||
output_pod(dfp, pod_yaml_file, node_select=False)
|
||||
print("OUTPUT: %s (len: %d)" % (pod_yaml_file, len(dfp)))
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Based on trace files in csv/, we will generate 23 trace folders (in parallel) that will be used as input folder for the simulator
|
||||
for file in csv/openb_pod_list*; do
|
||||
echo $file
|
||||
# Create trace folders with the name of csv trace file (without .csv)
|
||||
filename="${file##*/}"
|
||||
OUTDIR="${filename%.*}"
|
||||
mkdir -p $OUTDIR
|
||||
# Copy node yaml (generated by openb_node_list_gpu_node.csv) to the trace folder
|
||||
cp node_yaml/openb_node_list_gpu_node.yaml $OUTDIR/
|
||||
# Transform csv file to yaml file and put it under trace folder
|
||||
python3 pod_csv_to_yaml.py $file &
|
||||
done
|
||||
wait && date
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
# 1 Introduction
|
||||
|
||||
Understanding the microarchitectural resource characteristics of datacenter jobs has become increasingly critical to guarantee the performance of jobs while improving resource utilization. We provide a new open trace, AMTrace (Alibaba Microarchitecture Trace ), which is profiled from 8,577 high-end physical hosts from Alibaba’s datacenter by a hardware/software co-design monitoring method. AMTrace provides the microarchitectural metrics of 9.8 × 10^5 Linux containers with "per-container-per-logic CPU" granularity. AMTrace provides a new perspective to analyze the microarchitectural resource characteristics of datacenter jobs.
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
- [1 Introduction](#1-introduction)
|
||||
- [2 System Architecture](#2-system-architecture)
|
||||
- [3 Tables](#3-tables)
|
||||
- [core_pmu_metrics](#core_pmu_metrics)
|
||||
- [uncore_pmu_metrics](#uncore_pmu_metrics)
|
||||
- [container_meta](#container_meta)
|
||||
- [host_meta](#host_meta)
|
||||
- [4 Data](#4-data)
|
||||
- [5 Analysis Scripts](#5-analysis-scripts)
|
||||
|
||||
|
||||
# 2 System Architecture
|
||||
|
||||
|
||||
|
||||
Most LC jobs in Alibaba are Java-based e-commerce trading applications. They are containerized and scheduled by Kubernetes. Batch jobs are data processing batch jobs, such as Map-Reduce jobs, Spark jobs, and machine-learning training jobs. They are scheduled by Fuxi. To improve resource utilization, Alibaba uses a colocation architecture (shown in Figure 1), which can colocate LC jobs and Batch jobs on the same host and adopt several resource management technologies to mitigate resource contention.
|
||||
|
||||
To reduce CPU contention, we divide all logic CPUs into three groups: Batch CPUSet Pool, LC CPUSet Pool, and LC CPUShare Pool. The sizes of the three pools change dynamically with job scheduling. Jobs are scheduled to the corresponding groups according to their priority. The largest resource consumer, CPUShare Batch jobs, are allowed to utilize the resources in both LC CPUSet Pool and LC CPUShare Pool, but their resource priority is the lowest.
|
||||
|
||||
Other adopted resource isolation technologies, including CAT, BVT scheduling, and noise clean, are shown in our paper.
|
||||
|
||||
# 3 Tables
|
||||
|
||||
|
||||
## core_pmu_metrics
|
||||
|
||||
| Columns | Description | Type |Example Entry |
|
||||
|:-----------|:----------------|:--------|:---------------------------|
|
||||
| ts | Timestamp, the number of seconds from the start |Long |46309 |
|
||||
| node_id | ID of the node |String |e7a66b9189940e9d6102 |
|
||||
| container_id|ID of the container|String|213aff2b3dbec1f7c212|
|
||||
| cpu |logic cpu id|Int | 23|
|
||||
| core_id |core id |Int | 23|
|
||||
|socket_id |socket id|Int | 0|
|
||||
|instructions |number of instructions happened on the `logic cpu`|Long | 694470820|
|
||||
|cycles |number of cycles happened on the `logic cpu`|Long |1391935375|
|
||||
|ref_cycles|number of reference cycles happened on the `logic cpu` |Long|1284610800|
|
||||
|llc_misses |number of LLC cache misses happened on the `logic cpu` |Long|6667696|
|
||||
|
||||
|
||||
## uncore_pmu_metrics
|
||||
|
||||
| Columns | Description | Type |Example Entry |
|
||||
|:-----------|:----------------|:--------|:---------------------------|
|
||||
|ts | Timestamp, the number of seconds from the start|Long |2787 |
|
||||
|node_id |ID of the node | String| 68451b8967ad23a10681|
|
||||
|socket_id| socket id|Int| 0|
|
||||
|channel_id| ID of the memory channel| String | 4|
|
||||
|read_bw|read bandwidth (MiB/s)| Double| 2540.7185|
|
||||
|write_bw| write bandwidth (MiB/s)| Double| 1493.8240|
|
||||
|latency| memory read latency (ns)| Double| 16.7503|
|
||||
|
||||
## container_meta
|
||||
| Columns | Description | Type |Example Entry |
|
||||
|:-----------|:----------------|:--------|:---------------------------|
|
||||
|node_id|ID of the node|String|dd070834956a25c0c531|
|
||||
|container_id|ID of the container|String| a40de288d9455ba121db|
|
||||
|pod_id|ID of the Pod|String|bf74c065cd443d178474|
|
||||
|cpu_mode|cpu allocation mode|String| CPUShare|
|
||||
|app_name|the name of applicaton,Batch jobs are all "Batch"| String| Batch|
|
||||
|deploy_group| Deployment group, one application may have multiple deployment group| String| Batch|
|
||||
|container_type|container type | String| Batch-ops|
|
||||
|container_cpu_spec| **Deprecated** Ratio of logic CPUs requested by the container to the number of logic CPUs of the node (0-1)| Double| 0.04|
|
||||
|container_mem_spec| **Deprecated** Ratio of memory requested by the container to the memory of the node (0-1)|Double|0.003|
|
||||
|pod_cpu_spec|Ratio of logic CPUs requested by the Pod to the number of logic CPUs of the node (0-1)|Double|0.04|
|
||||
|pod_mem_spec|Ratio of memory requested by the Pod to the memory of the node (0-1)|Double|0.003|
|
||||
|
||||
`cpu_mode` is one of the following enumeration strings:
|
||||
|
||||
- CPUSet: CPUSet means the job will be pinned to several logic CPUs and will not share the logic CPUs with other LC jobs.
|
||||
- CPUShare: CPUShare means the job will be scheduled among dozens of logic CPUs but share the logic CPUs with other LC jobs.
|
||||
|
||||
`container_type` is one of the following enumeration stirngs:
|
||||
|
||||
- Batch-Set: Batch job that running in CPUSet mode.
|
||||
- Batch-Share:Batch job that running in CPUShare mode.
|
||||
- Batch-ops: The operations container of Batch jobs.
|
||||
- biz: The business container of LC jobs.
|
||||
- other: Some sidecar containers.
|
||||
- sys: System containers.
|
||||
|
||||
## host_meta
|
||||
|
||||
| Columns | Description | Type |Example Entry |
|
||||
|:-----------|:----------------|:--------|:---------------------------|
|
||||
|node_id|ID of the node|String| 3201ea36ad240dc51d6c|
|
||||
|cpu_num|The number of logic CPUs of the node|Int| 96 |
|
||||
|machine_model|The model of the node, one model is one kind of specification|String| machine-1|
|
||||
|cpu_model|The model of the CPU|String | skylake|
|
||||
|ref_freq_Ghz|The frequency of the CPU(Ghz)|Int|2.5|
|
||||
|dimms_per_channel|The number of DIMMs of every memory channel|Int|2|
|
||||
|
||||
|
||||
# 4 Data
|
||||
Consindering that the raw trace is pretty large (8,500 host data, more than 2TB), in this version, we sampled 1000-hosts data from the raw trace. The sampled trace data has the same distribution with the raw trace. The size of the sampled trace is 300 GB.
|
||||
|
||||
Before the downloading, please make sure that your disk have more than 300 GB available space.
|
||||
|
||||
Then, you can run the `get_data.sh` to download the data and use `gzip -d file_name` to unzip the files.
|
||||
|
||||
In the future version, we will find a lower-cost way to store/distribute the whole data.
|
||||
|
||||
# 5 Analysis Scripts
|
||||
Coming soon.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 793 KiB |
|
|
@ -1,45 +0,0 @@
|
|||
url='http://aliopentrace.oss-cn-beijing.aliyuncs.com/amtrace'
|
||||
mkdir data
|
||||
cd data
|
||||
|
||||
mkdir container_meta_1k
|
||||
mkdir host_meta_1k
|
||||
mkdir core_pmu_1k
|
||||
mkdir uncore_pmu_1k
|
||||
base_dir=$(pwd)
|
||||
|
||||
table_name="container_meta"
|
||||
folder_name="${table_name}_1k"
|
||||
cd $base_dir/$folder_name
|
||||
|
||||
for ((i = 0; i < 1; i++)); do
|
||||
command="curl ${url}/${folder_name}/${table_name}-${i}.csv.gz -o ${table_name}-${i}.csv.gz"
|
||||
${command}
|
||||
done
|
||||
|
||||
table_name="host_meta"
|
||||
folder_name="${table_name}_1k"
|
||||
cd $base_dir/$folder_name
|
||||
|
||||
for ((i = 0; i < 1; i++)); do
|
||||
command="curl ${url}/${folder_name}/${table_name}-${i}.csv.gz -o ${table_name}-${i}.csv.gz"
|
||||
${command}
|
||||
done
|
||||
|
||||
table_name="core_pmu"
|
||||
folder_name="${table_name}_1k"
|
||||
cd $base_dir/$folder_name
|
||||
|
||||
for ((i = 0; i < 108; i++)); do
|
||||
command="curl ${url}/${folder_name}/${table_name}-${i}.csv.gz -o ${table_name}-${i}.csv.gz"
|
||||
${command}
|
||||
done
|
||||
|
||||
table_name="uncore_pmu"
|
||||
folder_name="${table_name}_1k"
|
||||
cd $base_dir/$folder_name
|
||||
|
||||
for ((i = 0; i < 20; i++)); do
|
||||
command="curl ${url}/${folder_name}/${table_name}-${i}.csv.gz -o Node.tar.gz"
|
||||
${command}
|
||||
done
|
||||
Binary file not shown.
|
|
@ -1,140 +0,0 @@
|
|||
# Overview of Microservices Traces
|
||||
|
||||
The released traces contain the detailed runtime metrics of nearly twenty thousand microservices. They are collected from Alibaba production clusters of over ten thousand [bare-metal nodes](https://dl.acm.org/doi/10.1145/3373376.3378507) during twelve hours in 2021.
|
||||
|
||||
We conduct a characterization analysis on the trace in a paper, [Characterizing Microservice Dependency and Performance: Alibaba Trace Analysis](http://cloud.siat.ac.cn/pdca/socc2021-AlibabaTraceAnalysis.pdf), published in SoCC’21. And we would encourage anybody who uses this trace to cite our paper.
|
||||
|
||||
```BibTeX
|
||||
@inproceedings{luo2021characterizing,
|
||||
title={Characterizing Microservice Dependency and Performance: Alibaba Trace Analysis},
|
||||
author={Luo, Shutian and Xu, Huanle and Lu, Chengzhi and Ye, Kejiang and Xu, Guoyao and Zhang, Liping and Ding, Yu and He, Jian and Xu, Chengzhong},
|
||||
booktitle={Proceedings of the ACM Symposium on Cloud Computing},
|
||||
pages={412--426},
|
||||
year={2021}
|
||||
}
|
||||
```
|
||||
|
||||
# Cluster Architecture
|
||||
|
||||
The production cluster contains a large number of bare-metal (BM) nodes and is running in the Alibaba cloud. Users could submit their offline jobs through different controls, which will require resources from uniformed resource management (URM), or send microservices (MS) requirements to URM directly. To improve resource efficiency, URM will place containers of offline jobs and MS on the same BM node. More specifically, containers of offline jobs are scheduled into secure containers, which could be regarded as lightweight virtual machine. This can enhance isolation and mitigate interference from offline jobs to provide performance guarantees for MS containers.
|
||||
|
||||
|
||||
|
||||
# MS architecture in Alibaba
|
||||
|
||||
As shown in this figure, users usually send a web request to the entering MS first, e.g., MS A, which will trigger a series of calls between related microservices. We define the set of these calls as a call graph. As such, a call graph contains multiple calls between different pairs of microservices. Here, a pair of microservices contain one upstream microservice (UM) and one downstream microservice (DM).
|
||||
|
||||
|
||||
|
||||
Microservices can be categorized into two types, stateless services (e.g., a circle in the above Figure) and stateful services (e.g., a rectangle or hexagon). Stateless services are isolated from state data while stateful services need to store data in some locations, such as Database (DB) and Memcached (MC). There exist three types of communication paradigms between a pair of microservices, i.e., inter-process communication, remote invocation, and indirect communication.
|
||||
|
||||
# Introduction of Trace Data
|
||||
|
||||
The traces include four parts of data as follows:
|
||||
|
||||
node: BM Node runtime information. It records CPU and memory utilization of 1300+ BM nodes in a production cluster.
|
||||
|
||||
MS_Resource_Table: MS runtime information. It records CPU and memory utilization of 90000+ containers for 1300+ MSs in the same production cluster.
|
||||
|
||||
MS_Metrics_Table: Microservice call rate (MCR) and response time (RT) information. It records MCR and RT of the calls via different communication paradigms among 1300+ MSs with 90000+ containers in the same production cluster.
|
||||
|
||||
MS_CallGraph_Table: MS Call Graphs information. Due to the large-scale data size, we sample the call graph based on the rate of 0.5%. It contains about more than twenty million call graphs among 20000+ MSs with in more than ten clusters.
|
||||
|
||||
Users could run the following command to fetch data.
|
||||
|
||||
> bash fetchData.sh
|
||||
|
||||
It includes node/Node.tar.gz, MSCallGraph/MSCallGraph_*.tar.gz, MSResource/MSResource_*.tar.gz, MSRTQps/MSRTQps_*.tar.gz.
|
||||
|
||||
Size of each directory:
|
||||
|
||||
- 1.1Gi node
|
||||
- 25Gi MSCallGraph
|
||||
- 16Gi MSResource
|
||||
- 19Gi MSRTQps
|
||||
|
||||
Usage
|
||||
In each directory (node,MSCallGraph,MSResource,MSRTQps), please execute:
|
||||
|
||||
> for file in `ls *.tar.gz`; do tar -xzf $file; done
|
||||
|
||||
node:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------------ | :--------------------------------------------------------------: |
|
||||
| timestamp | 1000 |
|
||||
| nodeid | ff1fb31957db767c5be4de2855488f128532efc2df0a673c6fa3e7718d10f355 |
|
||||
| cpu_utilization | 0.7236219289416774 |
|
||||
| memory_utilization | 0.738378997619996 |
|
||||
|
||||
- timestamp: Timestamp of recorded metrics. Range from 0 to 43200000 for twelve hours (12 * 60 * 60 * 1000). The recording interval is the 30s (30 * 1000).
|
||||
- nodeid: The specific id of BM node. It could be joined with nodeid in MS_Resource_Table.
|
||||
- cpu_utilization: CPU utilization of BM node.
|
||||
- memory_utilization: Memory utilization of BM node.
|
||||
|
||||
MS_Metrics_Table:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------------ | :--------------------------------------------------------------: |
|
||||
| timestamp | 0 |
|
||||
| msname | 99f2e7b501f50db9b4089242a5d3e1aba334c32e0c718b8d79281529a9489b15 |
|
||||
| msinstanceid | 4d1cf65970444ef3ba9870468d7ecf9c17a93782134464767fbdc4aeb6f162eb |
|
||||
| nodeid | ecd8a876344d673d8e934f566aae38aec05e572d43ab8d58624bd59e3ec43928 |
|
||||
| cpu_utilization | 0.1299166666654249 |
|
||||
| memory_utilization | 0.6126489639282227 |
|
||||
|
||||
- timestamp: Mentioned in NodeTable. The recording interval is the 60s (60 * 1000).
|
||||
- msname: The name of MS, to be joined with MSName in MS_MCR_RT_Table, and DM and UM in MS_CallGraph_Table. MSName only contains stateless services, as stateful services run in other dedicated clusters.
|
||||
- msinstanceid: The specific container id of MS. An MS may have more than one container.
|
||||
- nodeid: The specific BM node in which MSInstanceID runs.
|
||||
- cpu_utilization: CPU utilization of MSInstanceID.
|
||||
- memory_utilization: Memory utilization of MSInstanceID.
|
||||
|
||||
MS_MCR_RT_Table:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------ | :--------------------------------------------------------------: |
|
||||
| timestamp | 6600000 |
|
||||
| msname | 1e5dd1f5843e50b9282fb99c58d8fe9c6e3d712d9e601a7a5264ca4ff7d96773 |
|
||||
| msinstanceid | 0c14b76dd7faa42a7e8f5fa5c59ad20e23ff8ce2981c1ff86544e76e72f1cca2 |
|
||||
| metrics | consumerRPC_RT |
|
||||
| value | 9.14142215173143 |
|
||||
|
||||
- timestamp: Mentioned in MS_Metrics_Table.
|
||||
- msname: Mentioned in MS_Metrics_Table.
|
||||
- msinstanceid: Mentioned in MS_Metrics_Table.
|
||||
- metrics: Calls Rate with different communication paradigms and corresponding RT. The value of metrics for an MS is an aggregation of all its DMs and UMs. To distinguish whether an MS is DM or UM, the Metrics are recorded with a prefix before communication paradigms. For example, RPC is named consumerRPC and providerRPC, meaning an MS as the consumer calling its DM and as the provider being called by its UM respectively. Correspondingly, MQ could be classified into two groups from an MS's point of view, namely, providerMQ, and consumerMQ. For the former, MQ is a provider that sends messages to the third party whereas, the latter is a consumer that fetches messages from the third party. As MSs in this table are all stateless services, they are only UMs to read or write stateful services.
|
||||
In summary, these metrics include consumerRPC_MCR, providerRPC_MCR, HTTP_MCR, providerMQ_MCR, consumerMQ_MCR, consumerRPC_RT, providerRPC_RT, HTTP_RT, providerMQ_RT, and consumerMQ_RT.
|
||||
- value: The value of Metrics. For example, the value of metric providerRPC_MCR and providerRPC_RT characterize the number of calls per second and the average of response time respectively. Here, the response time is measured by millisecond (ms).
|
||||
|
||||
MS_CallGraph_Table:
|
||||
|
||||
| columns | Example Entry |
|
||||
| --------- | :--------------------------------------------------------------: |
|
||||
| timestamp | 16397576 |
|
||||
| traceid | 015101cd15919399974329000e |
|
||||
| rpcid | 0.1.1.2.50 |
|
||||
| um | 35114acfb54c54fb9618f23cd28bbc57c765f597df140977d7030dcc52775ed4 |
|
||||
| rpctype | rpc |
|
||||
| interface | af42b5e3e0eb334d38619733586d78d1414f6549f24d31b39a5294454638bc59 |
|
||||
| dm | b65fdc9bfef6b4974c3e90e1ec7b92d30e639789da5a78c1d4685857e19c75a0 |
|
||||
| rt | 13 |
|
||||
|
||||
- timestamp: Mentioned in MS_CallGraph_Table.
|
||||
- traceid: Each call graph has a unique traceID.
|
||||
- rpcid: Each call is identified by a unique rpcID, which contains the ID information of a pair of UM and DM. For example, rpcID 0.1.1 and 0.1.2 denote two calls that two different DMs are called by the same UM, which is the DM in the call with rpcID 0.1. Note that, the call via remote invocation is recorded twice with the same rpcID in the UM and DM independently.
|
||||
- um: The name of UM.
|
||||
- rpctype: The communication paradigms. We record rpc_type as "DB" and "MC" for the calls via inter-process communication if DM is DB and MC respectively.
|
||||
- interface: The interface of DM is called by UM. The calls via remote invocation or HTTP have the interface.
|
||||
- DM: The name of DM.
|
||||
- rt: Response time of the call. It is measured by millisecond (ms). If rt is less than 1 ms, e.g. rt of read/write MC, the value will be recorded as 0. For call via RPC, the value of RT could be a positive integer and negative integer, which are recorded in UM and DM respectively, and represents UM RT (from UM sending a request to receiving a reply) and the opposite of DM RT (from DM receiving the request to sending the reply) respectively. The RT of a call via MQ is the interval from DM fetching the message to finishing it. For call via HTTP, the UM and DM RT is also recorded as positive integer and negative integer respectively.
|
||||
|
||||
# Discussion
|
||||
|
||||
- How to identify a specific service in the traces?
|
||||
|
||||
In practice, microservice architecture adopts proxy modules like Nginx to forward users' requests to an entering MS. The entering MS, e.g., interface of entering MS A in Fig.2, containing multiple interfaces, and each interface provides a specific service. As such, each interface of MS called by proxy MS could be labelled as an online service. As revealed in the paper, we classify the call graphs of each online service into different classes when analyzing the dynamics of microservices. It is worth noting that, some call graphs could even contain two proxy modules, and the name of the second proxy MS are usually recorded as '(?)' or ''.
|
||||
|
||||
- Missing items in traces.
|
||||
|
||||
In these traces, it happens that some metrics in MS_CallGraph_Table are lost. For example, the name of some MS is recorded as NAN, '(?)' or '' in the traces. As the call via RPC will be recorded twice in MS_CallGraph_Table, some metrics related to rpcID could be found from another record even if one is missing.
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
url='http://aliopentrace.oss-cn-beijing.aliyuncs.com/v2021MicroservicesTraces'
|
||||
|
||||
mkdir data
|
||||
cd data
|
||||
|
||||
mkdir Node
|
||||
mkdir MSResource
|
||||
mkdir MSRTQps
|
||||
mkdir MSCallGraph
|
||||
|
||||
cd Node
|
||||
|
||||
command="wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/node/Node_0.tar.gz"
|
||||
${command}
|
||||
|
||||
cd ../MSRTQps
|
||||
for((i=0;i<=24;i++));
|
||||
do
|
||||
command="wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/MSRTQps/MSRTQps_${i}.tar.gz"
|
||||
${command}
|
||||
done
|
||||
|
||||
cd ../MSCallGraph
|
||||
for((i=0;i<=144;i++));
|
||||
do
|
||||
command="wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/MSCallGraph/MSCallGraph_${i}.tar.gz"
|
||||
${command}
|
||||
done
|
||||
|
||||
cd ../MSResource
|
||||
for((i=0;i<=11;i++));
|
||||
do
|
||||
command="wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/MSResource/MSResource_${i}.tar.gz"
|
||||
${command}
|
||||
done
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 91 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
|
@ -1,157 +0,0 @@
|
|||
# Overview of Microservices Traces
|
||||
|
||||
The released traces contain the detailed runtime metrics of nearly twenty thousand microservices. They are collected from Alibaba production clusters of over ten thousand [bare-metal nodes](https://dl.acm.org/doi/10.1145/3373376.3378507) during 13 days in 2022. In comparison to the previous trace version (v2021), this updated trace offers an extended duration (13 days) and includes additional information, such as the service ID within the call graph.
|
||||
|
||||
```BibTeX
|
||||
@inproceedings{luo2022Prediction,
|
||||
title={The Power of Prediction: Microservice Auto Scaling via Workload Learning},
|
||||
author={Luo, Shutian and Xu, Huanle and Ye, Kejiang and Xu, Guoyao and Zhang, Liping and Yang, Guodong and Xu, Chengzhong},
|
||||
booktitle={Proceedings of the ACM Symposium on Cloud Computing},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
|
||||
# Cluster Architecture
|
||||
|
||||
The production cluster contains a large number of bare-metal (BM) nodes and is running in the Alibaba cloud. Users could submit their offline jobs through different controls, which will require resources from uniformed resource management (URM), or send microservices (MS) requirements to URM directly. To improve resource efficiency, URM will place containers of offline jobs and MS on the same BM node. More specifically, containers of offline jobs are scheduled into secure containers, which could be regarded as lightweight virtual machine. This can enhance isolation and mitigate interference from offline jobs to provide performance guarantees for MS containers.
|
||||
|
||||
|
||||
|
||||
# MS architecture in Alibaba
|
||||
|
||||
As shown in this figure, users usually send a web request to the entering MS first, e.g., MS A, which will trigger a series of calls between related microservices. We define the set of these calls as a call graph. As such, a call graph contains multiple calls between different pairs of microservices. Here, a pair of microservices contain one upstream microservice (UM) and one downstream microservice (DM).
|
||||
|
||||
|
||||
|
||||
Microservices can be categorized into two types, stateless services (e.g., a circle in the above Figure) and stateful services (e.g., a rectangle or hexagon). Stateless services are isolated from state data while stateful services need to store data in some locations, such as Database (DB) and Memcached (MC). There exist three types of communication paradigms between a pair of microservices, i.e., inter-process communication, remote invocation, and indirect communication.
|
||||
|
||||
# Overview of Trace Data
|
||||
|
||||
The traces include four parts of data as follows:
|
||||
|
||||
Node: BM Node runtime information. It records CPU and memory utilization of 40000+ BM nodes in a production cluster.
|
||||
|
||||
MSResource: MS runtime information. It records CPU and memory utilization of 470000+ containers for 28000+ MSs in the same production cluster.
|
||||
|
||||
MSRTMCR: Microservice call rate (MCR) and response time (RT) information. It records MCR and RT of the calls via different communication paradigms among 28000+ MSs with 470000+ containers in the same production cluster.
|
||||
|
||||
MSCallGraph: MS Call Graphs information. It contains about more than twenty million call graphs among 17000+ MSs with in more than ten clusters.
|
||||
|
||||
Note: The value of resource utilization and the MCR has been **normalized** by Max-min method.
|
||||
|
||||
# Trace Data Download
|
||||
|
||||
User can use the following script to download the trace with different intervals.
|
||||
|
||||
> bash fetchData.sh start_date=0d0 end_date=1d1
|
||||
|
||||
Where the `start_date` and `end_date` follow the following format: `${day}d${hour}`, and they are `[start_date, end_date)`. The day and hour are all started by 0, data will be saved in `data/MSCallGraph`, `data/MSResource`, `data/Node` and `data/MSRTMCR` respectively.
|
||||
|
||||
Size of each directory (compressed) for an hour:
|
||||
|
||||
- ~10Mi Node
|
||||
- ~4Gi MSCallGraph
|
||||
- ~700Mi MSResource
|
||||
- ~3Gi MSRTMCR
|
||||
|
||||
The size of all files for 13 days is about 2T.
|
||||
|
||||
Usage
|
||||
In each directory (Node,MSCallGraph,MSResource,MSRTMCR), please execute:
|
||||
|
||||
> for file in `ls *.tar.gz`; do tar -xzf $file; done
|
||||
|
||||
# Introduction of Trace Data
|
||||
|
||||
Node:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------------ | :---------------: |
|
||||
| timestamp | 60000 |
|
||||
| nodeid | NODE_10632 |
|
||||
| cpu_utilization | 0.266488095525847 |
|
||||
| memory_utilization | 0.159064258887333 |
|
||||
|
||||
- timestamp: Timestamp of recorded metrics. The recording interval is the 60s (60 * 1000).
|
||||
- nodeid: The specific id of BM node. It could be joined with nodeid in MS_Resource_Table.
|
||||
- cpu_utilization: **Normalized** CPU utilization of BM node.
|
||||
- memory_utilization: **Normalized** memory utilization of BM node.
|
||||
|
||||
MSResource:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------------ | :-----------------: |
|
||||
| timestamp | 180000 |
|
||||
| msname | MS_21881 |
|
||||
| msinstanceid | MS_21881_POD_0 |
|
||||
| nodeid | NODE_11517 |
|
||||
| cpu_utilization | 0.21995999999530616 |
|
||||
| memory_utilization | 0.833001454671224 |
|
||||
|
||||
- timestamp: Mentioned in Node. The recording interval is the 60s (60 * 1000).
|
||||
- msname: The name of MS, to be joined with MSName in MS_MCR_RT_Table, and DM and UM in MS_CallGraph_Table. MSName only contains stateless services, as stateful services run in other dedicated clusters.
|
||||
- msinstanceid: The specific container id of MS. An MS may have more than one container.
|
||||
- nodeid: The specific BM node in which MSInstanceID runs.
|
||||
- cpu_utilization: **Normalized** CPU utilization of MSInstanceID.
|
||||
- memory_utilization: **Normalized** memory utilization of MSInstanceID.
|
||||
|
||||
MSRTMCR:
|
||||
|
||||
| columns | Example Entry |
|
||||
| --------------- | :--------------------: |
|
||||
| timestamp | 60000 |
|
||||
| msname | MS_73317 |
|
||||
| msinstanceid | MS_73317_POD_1797 |
|
||||
| nodeid | NODE_3619 |
|
||||
| providerrpc_rt | 10.119451170298627 |
|
||||
| providerrpc_mcr | 1.216773932801612e-05 |
|
||||
| consumerrpc_rt | 9.996974281391829 |
|
||||
| consumerrpc_mcr | 7.169679436055699e-12 |
|
||||
| writemc_rt | 0.0 |
|
||||
| writemc_mcr | 0.0 |
|
||||
| readmc_rt | 0.40625 |
|
||||
| readmc_mcr | 3.142596113773332e-07 |
|
||||
| writedb_rt | 0.0 |
|
||||
| writedb_mcr | 0.0 |
|
||||
| readdb_rt | 0.9693548387096775 |
|
||||
| readdb_mcr | 6.088779970435831e-06 |
|
||||
| consumermq_rt | 0.0 |
|
||||
| consumermq_mcr | 0.0 |
|
||||
| providermq_rt | 24.4218009478673 |
|
||||
| providermq_mcr | 2.0721493125192907e-06 |
|
||||
| http_mcr | 0.0 |
|
||||
| http_rt | 0.0 |
|
||||
|
||||
- timestamp: Mentioned in Node. The recording interval is the 60s (60 * 1000).
|
||||
- msname: Mentioned in MSResource.
|
||||
- msinstanceid: Mentioned in MSResource.
|
||||
- nodeid: Mentioned in Node.
|
||||
- Other columns: The value of corresponding RT and calls rate with different communication paradigms. For example, the value of metric providerRPC_MCR and providerRPC_RT characterize the number of calls per second and the average of response time respectively. Here, the response time is measured by millisecond (ms) and the MCR is **normalized** through max-min in range from 0 to 1. The value of metrics for an MS is an aggregation of all its DMs and UMs. To distinguish whether an MS is DM or UM, the Metrics are recorded with a prefix before communication paradigms. For example, RPC is named consumerRPC and providerRPC, meaning an MS as the consumer calling its DM and as the provider being called by its UM respectively. Correspondingly, MQ could be classified into two groups from an MS's point of view, namely, providerMQ, and consumerMQ. For the former, MQ is a provider that sends messages to the third party whereas, the latter is a consumer that fetches messages from the third party. As MSs in this table are all stateless services, they are only UMs to read or write stateful services.
|
||||
In summary, these metrics include consumerRPC_MCR, providerRPC_MCR, HTTP_MCR, providerMQ_MCR, consumerMQ_MCR, consumerRPC_RT, providerRPC_RT, HTTP_RT, providerMQ_RT, and consumerMQ_RT.
|
||||
|
||||
MSCallGraph:
|
||||
|
||||
| columns | Example Entry |
|
||||
| ------------ | :--------------: |
|
||||
| timestamp | 115352 |
|
||||
| traceid | T_11560863075 |
|
||||
| service | S_153587416 |
|
||||
| rpc_id | 0.1 |
|
||||
| um | MS_58845 |
|
||||
| uminstanceid | MS_58845_POD_0 |
|
||||
| rpctype | rpc |
|
||||
| interface | xOuy6-80Vt |
|
||||
| dm | MS_71712 |
|
||||
| dminstanceid | MS_71712_POD_244 |
|
||||
| rt | 2.0 |
|
||||
|
||||
- timestamp: Mentioned in Node.
|
||||
- traceid: Each call graph has a unique traceID.
|
||||
- service: Online service id. A specific online service provides a function for users. For example, the online shopping application can provide multiple online services, including ordering, goods searching, delivering and so on.
|
||||
- rpcid: Each call is identified by a unique rpcID, which contains the ID information of a pair of UM and DM. For example, rpcID 0.1.1 and 0.1.2 denote two calls that two different DMs are called by the same UM, which is the DM in the call with rpcID 0.1. Note that, the call via remote invocation is recorded twice with the same rpcID in the UM and DM independently.
|
||||
- um: The name of UM.
|
||||
- uminstanceid: The specific container id of um MS. An MS may have more than one container.
|
||||
- rpctype: The communication paradigms. We record rpc_type as "DB" and "MC" for the calls via inter-process communication if DM is DB and MC respectively.
|
||||
- interface: The interface of DM is called by UM. The calls via remote invocation or HTTP have the interface.
|
||||
- dm: The name of DM.
|
||||
- dminstanceid: The specific container id of dm MS. An MS may have more than one container.
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
#!/bin/bash
|
||||
prepare_dir() {
|
||||
mkdir -p data/NodeMetrics data/MSMetrics data/MSRTMCR data/CallGraph
|
||||
}
|
||||
|
||||
# $1 = start_day, $2 = end_day
|
||||
# $3 = start_hour, $4 = end_hour
|
||||
fetch_data() {
|
||||
declare -a file_names=(
|
||||
"data/CallGraph/CallGraph" "data/MSMetrics/MSMetrics"
|
||||
"data/NodeMetrics/NodeMetrics" "data/MSRTMCR/MSRTMCR"
|
||||
)
|
||||
declare -a remote_paths=(
|
||||
"CallGraph/CallGraph" "MSMetricsUpdate/MSMetricsUpdate"
|
||||
"NodeMetricsUpdate/NodeMetricsUpdate" "MCRRTUpdate/MCRRTUpdate"
|
||||
)
|
||||
declare -a ratios=(3 30 720 3)
|
||||
start_hour=$(($1 * 24 * 60 + $3 * 60))
|
||||
end_hour=$(($2 * 24 * 60 + $4 * 60))
|
||||
for i in $(seq 0 3); do
|
||||
start_idx=$(($start_hour / ${ratios[$i]}))
|
||||
end_idx=$(($end_hour / ${ratios[$i]} - 1))
|
||||
if [[ $i == 2 && $(($end_hour % ${ratios[$i]})) != 0 ]]; then
|
||||
end_idx=$(($end_idx + 1))
|
||||
fi
|
||||
for idx in $(seq $start_idx $end_idx); do
|
||||
file_name="${file_names[$i]}_$idx.tar.gz"
|
||||
remote_path="${remote_paths[$i]}_$idx.tar.gz"
|
||||
url="https://aliopentrace.oss-cn-beijing.aliyuncs.com/v2022MicroservicesTraces/$remote_path"
|
||||
command="wget -c --retry-connrefused --tries=0 --timeout=50 -O $file_name $url"
|
||||
$command
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
for ARGUMENT in "$@"; do
|
||||
KEY=$(echo $ARGUMENT | cut -f1 -d=)
|
||||
|
||||
KEY_LENGTH=${#KEY}
|
||||
VALUE="${ARGUMENT:$KEY_LENGTH+1}"
|
||||
|
||||
export "$KEY"="$VALUE"
|
||||
done
|
||||
start_day=$(expr $(echo $start_date | cut -f1 -dd) + 0)
|
||||
start_hour=$(expr $(echo $start_date | cut -f2 -dd) + 0)
|
||||
end_day=$(expr $(echo $end_date | cut -f1 -dd) + 0)
|
||||
end_hour=$(expr $(echo $end_date | cut -f2 -dd) + 0)
|
||||
prepare_dir
|
||||
fetch_data $start_day $end_day $start_hour $end_hour
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 91 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 88 KiB |
|
|
@ -12,9 +12,7 @@ The format of trace data is described in the [schema description](trace_201708.m
|
|||
# Downloading the trace
|
||||
The data is stored in Alibaba Cloud Object Storage Service. You do not need to have an Alibaba account or sign up for Object Storage Service to download the data.
|
||||
|
||||
Users can run the following command to fetch data.
|
||||
>wget -c --retry-connrefused --tries=0 --timeout=50 http://aliopentrace.oss-cn-beijing.aliyuncs.com/v2017Traces/alibaba-trace-2017.tar.gz
|
||||
|
||||
Downloading information can be found (after a short survey) in [this link](https://goo.gl/forms/eOoe6DwZQpd2H5n53). We use the contact information to keep in touch with you, and announce goodies such as new traces.
|
||||
Included with the trace is a [SHA256SUM](SHA256SUM) file, which can be used to verify the integrity of a download, using the sha256sum command from GNU coreutils using a command like
|
||||
|
||||
```
|
||||
|
|
|
|||
|
|
@ -41,8 +41,8 @@ container_usage.csv,11, maximum cycles per instruction, FLOAT, NO
|
|||
container_usage.csv,12, maximum last-level cache misses per 1000 instructions, FLOAT, NO
|
||||
server_event.csv,1, timestamp, INTEGER, YES
|
||||
server_event.csv,2, machine id, INTEGER, YES
|
||||
server_event.csv,3, event type, STRING, YES
|
||||
server_event.csv,4, event detail, STRING, NO
|
||||
server_event.csv,3, event type, INTEGER, YES
|
||||
server_event.csv,4, event detail, INTEGER, NO
|
||||
server_event.csv,5, number of cpus, INTEGER, NO
|
||||
server_event.csv,6, normalized memory, FLOAT, NO
|
||||
server_event.csv,7, normalized disk space, FLOAT, NO
|
||||
|
|
|
|||
|
|
|
@ -1,13 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
url='http://aliopentrace.oss-cn-beijing.aliyuncs.com/v2018Traces'
|
||||
|
||||
mkdir data
|
||||
cd data
|
||||
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/machine_meta.tar.gz
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/machine_usage.tar.gz
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/container_meta.tar.gz
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/container_usage.tar.gz
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/batch_task.tar.gz
|
||||
wget -c --retry-connrefused --tries=0 --timeout=50 ${url}/batch_instance.tar.gz
|
||||
|
|
@ -5,99 +5,90 @@ This file describes the schema of each data file.
|
|||
The index below is aligned with the data column in each file.
|
||||
|
||||
### machine meta
|
||||
+-------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+-------------------------------------------------------------------------------------+
|
||||
| machine_id | string | | uid of machine |
|
||||
| time_stamp | bigint | | time stamp, in second |
|
||||
| failure_domain_1 | bigint | | one level of container failure domain |
|
||||
| failure_domain_2 | string | | another level of container failure domain |
|
||||
| cpu_num | bigint | | number of cpu on a machine |
|
||||
| mem_size | bigint | | normalized memory size. [0, 100] |
|
||||
| status | string | | status of a machine |
|
||||
+-------------------------------------------------------------------------------------+
|
||||
+------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+------------------------------------------------------------------------------------+
|
||||
| machine_id | string | | uid of machine |
|
||||
| time_stamp | bigint | | time stamp, in second |
|
||||
| failure_domain_1 | bigint | | one level of container failure domain |
|
||||
| failure_domain_2 | string | | another level of container failure domain |
|
||||
| cpu_num | bigint | | number of cpu on a machine |
|
||||
| mem_size | bigint | | normalized memory size. [0, 100] |
|
||||
| status | string | | status of a machine |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
||||
* about failure_domain_1: We have multiple levels of failure domains of which two are provided in this version of trace. For any application that requires fault tolerance, their instances should be spread across many failure domains. This is an enumerate value.
|
||||
|
||||
### machine usage
|
||||
+--------------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+--------------------------------------------------------------------------------------------+
|
||||
| machine_id | string | | uid of machine |
|
||||
| time_stamp | double | | time stamp, in second |
|
||||
| cpu_util_percent | bigint | | [0, 100] |
|
||||
| mem_util_percent | bigint | | [0, 100] |
|
||||
| mem_gps | double | | normalized memory bandwidth, [0, 100] |
|
||||
| mkpi | bigint | | cache miss per thousand instruction |
|
||||
| net_in | double | | normarlized in coming network traffic, [0, 100] |
|
||||
| net_out | double | | normarlized out going network traffic, [0, 100] |
|
||||
| disk_io_percent | double | | [0, 100], abnormal values are of -1 or 101 |
|
||||
+--------------------------------------------------------------------------------------------+
|
||||
+------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+------------------------------------------------------------------------------------+
|
||||
| machine_id | string | | uid of machine |
|
||||
| time_stamp | double | | time stamp, in second |
|
||||
| cpu_util_percent | bigint | | [0, 100] |
|
||||
| mem_util_percent | bigint | | [0, 100] |
|
||||
| mem_gps | double | | normalized memory bandwidth, [0, 100] |
|
||||
| mkpi | bigint | | cache miss per thousand instruction |
|
||||
| net_in | double | | normarlized in coming network traffic, [0, 100] |
|
||||
| net_out | double | | normarlized out going network traffic, [0, 100] |
|
||||
| disk_io_percent | double | | [0, 100], abnormal values are of -1 or 101 |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
||||
|
||||
### container meta
|
||||
+-----------------------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+-----------------------------------------------------------------------------------------------------+
|
||||
| container_id | string | | uid of a container |
|
||||
| machine_id | string | | uid of container's host machine |
|
||||
| time_stamp | bigint | | |
|
||||
+------------------------------------------------------------------------------------+
|
||||
| Field | Type | Label | Comment |
|
||||
+------------------------------------------------------------------------------------+
|
||||
| container_id | string | | uid of a container |
|
||||
| machine_id | string | | uid of container's host machine |
|
||||
| time_stamp | bigint | | |
|
||||
| app_du | string | | containers with same app_du belong to same application group |
|
||||
| status | string | | |
|
||||
| cpu_request | bigint | | 100 is 1 core |
|
||||
| cpu_limit | bigint | | 100 is 1 core |
|
||||
| mem_size | double | | normarlized memory, [0, 100] |
|
||||
+-----------------------------------------------------------------------------------------------------+
|
||||
|
||||
* about app_du: Containers belong to the same deploy unit provides one service, typically, they should be spread across failure domains
|
||||
| status | string | | |
|
||||
| cpu_request | bigint | | 100 is 1 core |
|
||||
| cpu_limit | bigint | | 100 is 1 core |
|
||||
| mem_size | double | | normarlized memory |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
||||
### container usage
|
||||
+-----------------------------------------------------------------------------------------+
|
||||
| container_id | string | | uid of a container |
|
||||
| machine_id | string | | uid of container's host machine |
|
||||
| time_stamp | double | | time stamp, in second |
|
||||
| cpu_util_percent | bigint | | |
|
||||
| mem_util_percent | bigint | | |
|
||||
| cpi | double | | |
|
||||
| mem_gps | double | | normalized memory bandwidth, [0, 100] |
|
||||
| mpki | bigint | | |
|
||||
| net_in | double | | normarlized in coming network traffic, [0, 100] |
|
||||
| net_out | double | | normarlized out going network traffic, [0, 100] |
|
||||
| disk_io_percent | double | | [0, 100], abnormal values are of -1 or 101 |
|
||||
+-----------------------------------------------------------------------------------------+
|
||||
+------------------------------------------------------------------------------------+
|
||||
| container_id | string | | uid of a container |
|
||||
| machine_id | string | | uid of container's host machine |
|
||||
| time_stamp | double | | time stamp, in second |
|
||||
| cpu_util_percent | bigint | | |
|
||||
| mem_util_percent | bigint | | |
|
||||
| cpi | double | | |
|
||||
| mem_gps | double | | normalized memory bandwidth, [0, 100] |
|
||||
| mpki | bigint | | |
|
||||
| net_in | double | | normarlized in coming network traffic, [0, 100] |
|
||||
| net_out | double | | normarlized out going network traffic, [0, 100] |
|
||||
| disk_io_percent | double | | [0, 100], abnormal values are of -1 or 101 |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
||||
### batch task
|
||||
+----------------------------------------------------------------------------------------+
|
||||
| task_name | string | | task name. unique within a job |
|
||||
| instance_num | bigint | | number of instances |
|
||||
| job_name | string | | job name |
|
||||
| task_type | string | | task type |
|
||||
| status | string | | task status |
|
||||
| start_time | bigint | | start time of the task |
|
||||
| end_time | bigint | | end of time the task |
|
||||
+------------------------------------------------------------------------------------+
|
||||
| task_name | string | | task name. unique within a job |
|
||||
| instance_num | bigint | | number of instances |
|
||||
| job_name | string | | job name |
|
||||
| task_type | string | | task type |
|
||||
| status | string | | task status |
|
||||
| start_time | bigint | | start time of the task |
|
||||
| end_time | bigint | | end of time the task |
|
||||
| plan_cpu | double | | number of cpu needed by the task, 100 is 1 core |
|
||||
| plan_mem | double | | normalized memorty size, [0, 100] |
|
||||
+----------------------------------------------------------------------------------------+
|
||||
|
||||
* task name indicates the DAG information, see the explanation of batch workloads
|
||||
| plan_mem | double | | normalized memorty size |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
||||
### batch instance
|
||||
+-----------------------------------------------------------------------------------------------+
|
||||
| instance_name | string | | instance name of the instance |
|
||||
| task_name | string | | name of task to which the instance belong |
|
||||
| job_name | string | | name of job to which the instance belong |
|
||||
| task_type | string | | task type |
|
||||
| status | string | | instance status |
|
||||
| start_time | bigint | | start time of the instance |
|
||||
| end_time | bigint | | end time of the instance |
|
||||
| machine_id | string | | uid of host machine of the instance |
|
||||
| seq_no | bigint | | sequence number of this instance |
|
||||
| total_seq_no | bigint | | total sequence number of this instance |
|
||||
| cpu_avg | double | | average cpu used by the instance, 100 is 1 core |
|
||||
| cpu_max | double | | max cpu used by the instance, 100 is 1 core |
|
||||
| mem_avg | double | | average memory used by the instance (normalized) |
|
||||
| mem_max | double | | max memory used by the instance (normalized, [0, 100]) |
|
||||
+-----------------------------------------------------------------------------------------------+
|
||||
|
||||
* Task name is uniqe within a job; note task name indicates the DAG information, see the explanation of batch workloads
|
||||
* There are totally 12 types, and only some of them have DAG info
|
||||
| instance_name | string | | instance name of the instance |
|
||||
| task_name | string | | name of task to which the instance belong |
|
||||
| job_name | string | | name of job to which the instance belong |
|
||||
| task_type | string | | task type |
|
||||
| status | string | | instance status |
|
||||
| start_time | bigint | | start time of the instance |
|
||||
| end_time | bigint | | end time of the instance |
|
||||
| machine_id | string | | uid of host machine of the instance |
|
||||
| seq_no | bigint | | sequence number of this instance |
|
||||
| total_seq_no | bigint | | total sequence number of this instance |
|
||||
| cpu_avg | double | | average cpu used by the instance, 100 is 1 core |
|
||||
| cpu_max | double | | average memory used by the instance (normalized) |
|
||||
| mem_avg | double | | max cpu used by the instance, 100 is 1 core |
|
||||
| mem_max | double | | max memory used by the instance (normalized) |
|
||||
+------------------------------------------------------------------------------------+
|
||||
|
|
|
|||
|
|
@ -60,16 +60,106 @@ total 98G
|
|||
|
||||
## 2.2 Schema
|
||||
|
||||
For the sake of clearity, all scheme description is moved to [schema](./schema.txt).
|
||||
|
||||
**Some explanation of common fields.**
|
||||
|
||||
* time_stamp, start_time and end_time: These fields in the tables are all with unit "second" and the number is the difference of between the actual time and the beginning of the period over which the trace is sampled. The beginning of period is 0.
|
||||
* time_stamp, start_time and end_time: These fields in the tables are all with unit "second" and the number if the difference of between the actual time and the beginning of the period over which the trace is sampled. The beginning of period is 0.
|
||||
* For confidential reason, we normalize some values such as memory size and disk size and rescale such field between [0, 100]. However, there are some invalid values and they are set to either -1 or 101.
|
||||
|
||||
## 2.3 DAG of batch workloads
|
||||
**Explanation of tables.**
|
||||
|
||||
*In this version of cluster data, we include many types of batch workloads. Most of them are DAGs while some of them are not. For those tasks that are not DAGs, we name them using random characters, such as `task_Nzg3ODAwNDgzMTAwNTc2NTQ2Mw==` or `task_Nzg3ODAwNDgzMTAwODc2NTQ3MQ==`. These tasks can be treated as independent tasks. In the remainder of this section, we explain how to deduce DAG of a job from the task names.*
|
||||
* machine_meta.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| machine_id | string | The unique ID (uid) of a machine | |
|
||||
| time_stamp | int | time index | 0 means the time stamp is before or after the 8-day time span |
|
||||
| disaster_level_1 | enum | 1st level of failure domain | We have multiple levels of failure domains of which two are provided in this version of trace. For any application that requires fault tolerance, their instances should be spread across many failure domains |
|
||||
| disaster_level_2 | enum | 2nd level of failure domain | Similar to disaster_level_1, this is just another level of failure domain |
|
||||
| cpu_num | int | num of cpu of a machine | Unit is "core", e.g. 4 means 4 cores on the machine |
|
||||
| mem_size | int | memory size of a machine | Normalized to the largest memory size of all machines |
|
||||
| status | enum | status of a machine at given time_stamp | Status of the machine. See explanation below |
|
||||
|
||||
* machine_usage.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| machine_id | string | The unique ID (uid) of a machine | |
|
||||
| time_stamp | int | time stamp | 0 means the time stamp is before or after the 8-day time span |
|
||||
| cpu_util_percent | int | cpu utilization percentage | Both user and system are included, between [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| mem_util_percent | int | memory utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| mem_gps | int | memory bandwidth usage | Normalized to maximum memory bandwith usage of all machines |
|
||||
| mpki | int | cache miss per thousand instruction | |
|
||||
| net_in | int | number of incoming network packages | Normalized to the maximum of this column |
|
||||
| net_out | int | number of outgoing network packages | Normalized to the maximum of this column |
|
||||
| disk_usage_percent | int | disk space utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| disk_io_percent | int | disk io utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
|
||||
* container_meta.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| container_id | string | The unique ID (uid) of a container | |
|
||||
| machine_id | string | machine uid of a container's host | |
|
||||
| deploy_unit | string | deploy group of a container | Containers belong to the same deploy unit provides one service, typically, they should be spread across failure domains |
|
||||
| time_stamp | int | time stamp | 0 means the time stamp is before or after the 8-day time span |
|
||||
| cpu_request | int | planned cpushare request | 100 means 1 core |
|
||||
| cpu_limit | int | planned cpushare request | 100 means 1 core |
|
||||
| mem_size | int | planned memory size | Normalized to the largest memory size of all machines |
|
||||
| status | enum | status of a container at given time_stamp | Status of a container at given time stamp, see the state machine for details |
|
||||
|
||||
* container_usage.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| container_id | string | The unique ID (uid) of a container | |
|
||||
| machine_id | string | machine uid on which a container runs | |
|
||||
| time_stamp | int | time stamp | 0 means the time stamp is before or after the 8-day time span |
|
||||
| cpu_util_percent | int | cpu utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| mpki | int | cache miss per thousand instruction | |
|
||||
| cpi | int | cpi of a container at given time_stamp | |
|
||||
| mem_util_percent | int | memory utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| mem_gps | int | memory bandwidth usage | Normalized to maximum memory bandwith usage of all machines |
|
||||
| disk_usage_percent | int | disk space utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| disk_io_percent | int | disk io utilization percentage | [0, 100]. There are some invalid values and they are set to -1 or 101 |
|
||||
| net_in | int | number of incoming network packages | Normalized to the largest net_in of all machines |
|
||||
| net_out | int | number of outgoing network packages | Normalized to the largest net_out of all machines |
|
||||
|
||||
* batch_instance.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| inst_name | string | The unique ID (uid) of a batch instance | Instance name is unique within a (job, task) pair |
|
||||
| task_name | string | The task name to which an instance belongs | Task name is uniqe within a job; note task name indicates the DAG information, see the explanation of batch workloads |
|
||||
| task_type | enum | type of the task | There are totally 12 types, and only some of them have DAG info |
|
||||
| job_name | string | job_name of the task that an instance belongs to | A job is consisted of many tasks. See the explanation of job-task-instance |
|
||||
| status | enum | status of an instance | Status of the instance |
|
||||
| start_time | int | start time of an instance | 0 means the time stamp is before or after the 8-day time span |
|
||||
| end_time | int | end time of an instance | 0 means the time stamp is before or after the 8-day time span |
|
||||
| machine_id | string | the machine id on which an instance runs | |
|
||||
| seq_no | int | seq no of an instance | |
|
||||
| total_seq_no | int | total seq no of an instance | |
|
||||
| cpu_avg | int | average cpu utilization of cpu of an instance | 100 means 1 core |
|
||||
| cpu_max | int | max cpu utilization of cpu of an instance | 100 means 1 core |
|
||||
| mem_avg | int | average memory utilization of cpu of an instance | Normalized to the largest memory size of all machines |
|
||||
| mem_max | int | max memory utilization of cpu of an instance | Normalized to the largest memory size of all machines |
|
||||
|
||||
* batch_task.csv
|
||||
|
||||
| column name | type | explanation | comments |
|
||||
| -------- | -------- | -------- | -------- |
|
||||
| task_name | string | The task name to which an instance belongs | Note task name indicates the DAG information, see the explanation of batch workloads |
|
||||
| inst_num | int | number of instances a task has | |
|
||||
| task_type | enum | type of the task | |
|
||||
| job_name | string | job_name of a task | |
|
||||
| status | enum | status of an instance | status of the task |
|
||||
| start_time | int | start time of an instance | 0 means the time stamp is before or after the 8-day time span |
|
||||
| end_time | int | end time of an instance | 0 means the time stamp is before or after the 8-day time span |
|
||||
| plan_cpu | int | cpu requested for each instance of the task | 100 means 1 core |
|
||||
| plan_mem | int | normalized memory requested for each instance of the task | Normalized to the largest memory size of all machines |
|
||||
|
||||
## 2.3 DAG of batch worloads
|
||||
|
||||
*In this version of cluster data, we include many types of batch wokrloads. Most of them are DAGs while some of them are not. For those tasks that are not DAGs, we name them using random characters, such as `task_Nzg3ODAwNDgzMTAwNTc2NTQ2Mw==` or `task_Nzg3ODAwNDgzMTAwODc2NTQ3MQ==`. These tasks can be treated as independent tasks. In the remainder of this section, we explain how to deduce DAG of a job from the task names.*
|
||||
|
||||
A complete batch computation job can be described using "job-task-instance" model. We will describe the meaning of each term and how the DAG information is expressed in the trace.
|
||||
|
||||
|
|
@ -99,13 +189,14 @@ The number of instances for each task is expressed with another field `instance_
|
|||
|
||||
If you do not want to do the survey ([survey link](http://alibabadeveloper.mikecrm.com/BdJtacN)), you could have the trace too.
|
||||
|
||||
The entire data set is about 48GB in .tar.gz format and 280GB after extraction. Due to [issue #49](https://github.com/alibaba/clusterdata/issues/49), we provide two download links, one for Chinese users and other for oversea users, and you can choose whichever suits you.
|
||||
The entire data set is about 48GB in .tar.gz format and 280GB after extraction. So we provide two download links.
|
||||
|
||||
## Download
|
||||
|
||||
Users can run the following command to fetch 6 files.
|
||||
> bash fetchData.sh
|
||||
|
||||
# 5 Acknowledgement
|
||||
|
||||
This version of cluster data would not have been well prepared without help from several interns, Jing Guo, Zihao Chang, Kangjing Wang, and Guoyao Xu, etc. Their efforts in data collection, verification, and document writing are very much appreciated.
|
||||
* Download the all as one dataset:
|
||||
* [alibaba_clusterdata_v2018](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/alibaba_clusterdata2018.tar.gz)
|
||||
* Download 6 files separately:
|
||||
* [machine_meta](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/machine_meta.tar.gz)
|
||||
* [machine_usage](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/machine_usage.tar.gz)
|
||||
* [container_meta](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/container_meta.tar.gz)
|
||||
* [container_usage](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/container_usage.tar.gz)
|
||||
* [batch_task](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/batch_task.tar.gz)
|
||||
* [batch_instance](http://clusterdata2018pubcn.oss-cn-beijing.aliyuncs.com/batch_instance.tar.gz)
|
||||
|
|
|
|||
Loading…
Reference in New Issue