baohan
/
clusterdata
0
0
Fork 1
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update ATC '23 paper to README

This commit is contained in:
qzweng 2023-06-05 15:41:56 +08:00
parent 50a91c7c74
commit 969ad9b4e6
1 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -8,9 +8,10 @@ So far, four versions of traces have been released:
* *cluster-trace-v2017* includes about 1300 machines in a period of 12 hours. The trace-v2017 firstly introduces the collocation of online services (aka long running applications) and batch workloads. To see more about this trace, see related documents ([trace_2017](./cluster-trace-v2017/trace_201708.md)). Download link is available after a short survey ([survey link](https://goo.gl/forms/eOoe6DwZQpd2H5n53)).
* *cluster-trace-v2018* includes about 4000 machines in a period of 8 days. Besides having larger scaler than trace-v2017, this piece trace also contains the DAG information of our production batch workloads. See related documents for more details ([trace_2018](./cluster-trace-v2018/trace_2018.md)). Download link is available after a survey (less than a minute, [survey link](http://alibabadeveloper.mikecrm.com/BdJtacN)).
* *cluster-trace-gpu-v2020* includes over 6500 GPUs (on ~1800 machines) in a period of 2 months. It describe the AI/ML workloads in the MLaaS (Machine-Learning-as-a-Service) provided by the [Alibaba PAI (Platform for Artificial Intelligence)](https://www.alibabacloud.com/product/machine-learning) on GPU clusters. See the subdirectory ([pai_gpu_trace_2020](./cluster-trace-gpu-v2020/README.md)) for the released data, schema, and scripts for processing and visualization. Our analysis paper, published in NSDI '22, is also available [here](https://home.cse.ust.hk/~weiwa/papers/weng-nsdi22.pdf).
* *cluster-trace-gpu-v2020* includes over 6500 GPUs (on ~1800 machines) in a period of 2 months. It describes the AI/ML workloads in the MLaaS (Machine-Learning-as-a-Service) provided by the [Alibaba PAI (Platform for Artificial Intelligence)](https://www.alibabacloud.com/product/machine-learning) on GPU clusters. See the subdirectory ([pai_gpu_trace_2020](./cluster-trace-gpu-v2020/README.md)) for the released data, schema, and scripts for processing and visualization. Our analysis paper published in USENIX NSDI '22 is available [here](https://www.usenix.org/conference/nsdi22/presentation/weng).
* *cluster-trace-microservices-v2021* contains 20000+ microservices in a period of 12 hours. The traces the first released to introduce the runtime metrics of microservices in the production cluster, including call dependencies, respond time, call rates, and so on. See the subdirectory ([trace_2021](./cluster-trace-microservices-v2021/README.md)) for more details. Our analysis paper, accepted by SoCC '21, is available [here](http://cloud.siat.ac.cn/pdca/socc2021-AlibabaTraceAnalysis.pdf).
* *cluster-trace-microarchitecture-v2022* first provides AMTrace (Alibaba Microarchitecutre Trace). AMTrace is the first fine-granulairty and large-scale microarchitectural metrics of Alibaba Colocation Datacenter. Based AMTrace, researchers can analysis: CPU performance, microarchitecture contention, memory bandwidth contention and so on. [Our paper](https://doi.org/10.1145/3545008.3545026) is accepted by ICPP'22. See the subdirectory ([trace_2022](./cluster-trace-microarchitecture-v2022/README.md)) for more details.
* *cluster-trace-gpu-v2023* includes over 6200 GPUs (on ~1200 machines). It describes the AI/ML workloads with diverse resource specifications in a heterogeneous GPU cluster. In our "[Beware of Fragmentation](https://www.usenix.org/conference/atc23/presentation/weng)" paper (published in USENIX ATC '23), we modeled this trace in a [Kubernetes Scheduler Simulator](https://github.com/hkust-adsl/kubernetes-scheduler-simulator) and demonstrated that our proposed Fragmentation Gradient Descent (FGD) policy outperforms classic scheduling policies like Best-Fit, Dot-Product, etc. See [fgd_gpu_trace_2023](./cluster-trace-gpu-v2023/README.md) for the released data, schema, and scripts for processing.
We encourage anyone to use the traces for study or research purposes, and if you had any question when using the trace, please contact us via email: [alibaba-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com), or file an issue on Github. Filing an issue is recommanded as the discussion would help all the community. Note that the more clearly you ask the question, the more likely you would get a clear answer.
@ -37,13 +38,15 @@ Last but not least, we are always open to work together with researchers to impr
The fundamental idea of our releasing cluster data is to enable researchers & practitioners doing resaerch, simulation with more realistic data and thus making the result closer to industry adoption. It is a huge encouragement to us to see more works using our data. Here is a list of existing works using Alibaba cluster data. **If your paper uses our trace, it would be great if you let us know by sending us email** ([aliababa-clusterdata](mailto:alibaba-clusterdata@list.alibaba-inc.com)).
* cluster trace GPU v2023
* [Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent](https://www.usenix.org/conference/atc23/presentation/weng), Qizhen Weng* and Lingyun Yang* (co-first author), Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. In the 2023 USENIX Annual Technical Conference (ATC '23), Boston, MA, USA, July 2023.
* microarchitecture trace v2022
* [Characterizing Job Microarchitectural Profiles at Alibaba Scale: Dataset and Analysis](https://doi.org/10.1145/3545008.3545026), Kangjin Wang, Ying Li, Cheng Wang, Tong Jia, Kingsum Chow, Yang Wen, Yaoyong Dou, Guoyao Xu, Chuanjia Hou, Jie Yao, and Liping Zhang. In 51st International Conference on Parallel Processing (ICPP ’22), August 29-September 1, 2022, Bordeaux, France. ACM, New York, NY, USA, 11 pages.
* microservices trace v2021
* [Characterizing Microservice Dependency and Performance: Alibaba Trace Analysis](http://cloud.siat.ac.cn/pdca/socc2021-AlibabaTraceAnalysis.pdf), Shutian Luo, Huanle Xu, Chengzhi Lu, Kejiang Ye, Guoyao Xu, Liping Zhang, Yu Ding, Jian He, Chengzhong Xu. SoCC'21
* [μBench: an open-source factory of benchmark microservice applications](http://netgroup.uniroma2.it/Andrea_Detti/papers/journals/Bench_an_open-source_factory_of_benchmark_microservice_applications.pdf), A. Detti, L. Funari and L. Petrucci, IEEE Transactions on Parallel and Distributed Systems
* cluster trace GPU v2020
* [MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters](https://home.cse.ust.hk/~weiwa/papers/weng-nsdi22.pdf), by Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding, to appear in the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’22), Renton, WA, April 2022.
* [MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters](https://www.usenix.org/conference/nsdi22/presentation/weng), Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. In the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22), Renton, WA, USA, April 2022.
* cluster trace v2018
* [Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces](https://dl.acm.org/citation.cfm?doid=3326285.3329074), Jing Guo, Zihao Chang, Sa Wang, Haiyang Ding, Yihui Feng, Liang Mao, Yungang Bao, IEEE/ACM International Symposium on Quality of Service, IWQoS 2019
* [DeepJS: Job Scheduling Based on Deep Reinforcement Learning in Cloud Data Center](https://github.com/RobertLexis/CloudSimPy/blob/master/playground/paper/F0049-4.19.pdf), by Fengcun Li and Bo Hu.