Go to file
Bala 8ab9ef14a0
Update tools to build packages from hydrated dependencies (#1958)
* Update tools to build packages with dependencies hydrated

Hydrated build is a new approach where all the dependencies are hydrated from pre-built RPMs. This provides two advantages.
 1. Faster build. As no package need to wait for its dependency's build to complete, all packages will be fired to build in parallel. (Practically with slight delay as firing of pre-built nodes takes place before).
 2. Failure of any dependency build will not cause failure to the current package, as the dependency is fetched from cached RPMs.

Introduced a new flag HYDRATED_BUILD to enable this feature. Also it is pre-requisite to run `make hydrate-toolchain` and `make hydrate-rpms` with appropriate tar files before using this option.

This is achieved by replacing the BUILD to RUN node edge with BUILD to PREBUILT node edge (clone of the RUN node) nodes (if there is a corresponding RPM hydrated already).

* Fix go fmt check failures

* Fix typos

* Introduce graphscrubber

graphscrubber is for preprocessing the build graph to scheduler.
Currently it is only doing hydrated build related work. Ideally all the
graph optimization related jobs to be moved from scheduler to
graphscrubber.

* go fmt tidy

* Remove unnecessary functions from graphscrubber

* Fix Make rule to build graphscrubber before using it

* Use inline hydrated-build flag

* Update documentaion for HYDRATED_BUILD flag

* Update changes recommended in PR discussion

* go tidy

* Remove unnecessary arguments from graphPreprocessor

* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go

Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>

* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go

Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>

* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go

Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>

* retrugger checks

* Update toolkit/scripts/pkggen.mk

Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>

* Update toolkit/scripts/pkggen.mk

Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>

* Update toolkit/scripts/pkggen.mk

Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>

* Update toolkit/scripts/pkggen.mk

Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>

* retrugger checks

* retrugger checks

Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
2022-02-01 09:36:00 +05:30
.github Fix grubby build with newer versions of RPM (#2036) 2022-01-28 12:22:09 -08:00
LICENSES-AND-NOTICES Update licenses for the new specs 2020-08-19 01:48:59 +00:00
SPECS [main] Fixing circular dependencies in `SPECS-EXTENDED`. (#2048) 2022-01-31 17:18:41 -08:00
SPECS-EXTENDED [main] Fixing circular dependencies in `SPECS-EXTENDED`. (#2048) 2022-01-31 17:18:41 -08:00
SPECS-SIGNED Include KDB frontend for kgdb (CONFIG_KGDB_KDB) (#2043) 2022-01-31 09:43:33 -08:00
toolkit Update tools to build packages from hydrated dependencies (#1958) 2022-02-01 09:36:00 +05:30
.gitignore [main extended] Enable libguestfs (#1970) 2022-01-21 11:26:39 -08:00
CODE_OF_CONDUCT.md Initial CODE_OF_CONDUCT.md commit 2020-07-22 12:27:23 -07:00
CONTRIBUTING.md Initial CBL-Mariner commit to GitHub 2020-08-06 20:17:52 -07:00
LICENSE Initial LICENSE commit 2020-07-22 12:27:24 -07:00
README.md Add status badge, update quickstart workflow (#377) 2020-11-13 14:40:30 -08:00
SECURITY.md Initial SECURITY.md commit 2020-07-22 12:27:25 -07:00
SUPPORT.md Merge 1.0 to dev branch 2021-08-19 13:46:51 -07:00
cgmanifest.json [main] Fixing circular dependencies in `SPECS-EXTENDED`. (#2048) 2022-01-31 17:18:41 -08:00

README.md

CBL-Mariner

Release Branch Status
1.0 1.0 Status

CBL-Mariner is an internal Linux distribution for Microsofts cloud infrastructure and edge products and services. CBL-Mariner is designed to provide a consistent platform for these devices and services and will enhance Microsofts ability to stay current on Linux updates. This initiative is part of Microsofts increasing investment in a wide range of Linux technologies, such as SONiC, Azure Sphere OS and Windows Subsystem for Linux (WSL). CBL-Mariner is being shared publicly as part of Microsofts commitment to Open Source and to contribute back to the Linux community. CBL-Mariner does not change our approach or commitment to any existing third-party Linux distribution offerings.

CBL-Mariner has been engineered with the notion that a small common core set of packages can address the universal needs of first party cloud and edge services while allowing individual teams to layer additional packages on top of the common core to produce images for their workloads. This is made possible by a simple build system that enables:

  • Package Generation: This produces the desired set of RPM packages from SPEC files and source files.
  • Image Generation: This produces the desired image artifacts like ISOs or VHDs from a given set of packages.

Whether deployed as a container or a container host, CBL-Mariner consumes limited disk and memory resources. The lightweight characteristics of CBL-Mariner also provides faster boot times and a minimal attack surface. By focusing the features in the core image to just what is needed for our internal cloud customers there are fewer services to load, and fewer attack vectors.

When security vulnerabilities arise, CBL-Mariner supports both a package-based update model and an image based update model. Leveraging the common RPM Package Manager system, CBL-Mariner makes the latest security patches and fixes available for download with the goal of fast turn-around times.

Getting Started with CBL-Mariner:

Instructions for building CBL-Mariner may be found here: Toolkit Documentation

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Acknowledgments

Any Linux distribution, including CBL-Mariner, benefits from contributions by the open software community. We gratefully acknowledge all contributions made from the broader open source community, in particular:

  1. The Photon OS Project for SPEC files originating from the Photon distribution.

  2. The Fedora Project for SPEC files, particularly with respect to QT, DNF and several of their dependencies.

  3. GNU and the Free Software Foundation

  4. Linux from Scratch

  5. Openmamba for SPEC files