yunikorn-release
Apache YuniKorn (Incubating) - A Universal Scheduler
Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.
YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn’s architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.
Architecture
Following chart illustrates the high-level architecture of YuniKorn.
YuniKorn consists of the following components spread over multiple code repositories, most of the following projects are written in GoLang.
- Scheduler core: the brain of the scheduler, which makes placement decisions (Allocate container X on node Y) according to pre configured policies. See more in current repo yunikorn-core. Scheduler core is implemented in a way to be agnostic to scheduler implementation.
- Scheduler interface: the common scheduler interface used by shims and the core scheduler. Contains the API layer (with GRPC/programming language bindings) which is agnostic to container orchestrator systems like YARN/K8s. See more in yunikorn-scheduler-interface.
-
Resource Manager shims: allow container orchestrator systems talks to yunikorn-core through scheduler-interface. Which can be configured on existing clusters without code change.
Currently, yunikorn-k8shim is available for Kubernetes integration. Supporting other Resource Manager is our next priority.
- Scheduler User Interface: the YuniKorn web interface for app/queue management. See more in yunikorn-web.
Feature highlights
Main scheduling features:
- Features to support both batch jobs and long-running/stateful services
- Hierarchy queues with min/max resource quotas.
- Resource fairness between queues, users and apps.
- Cross-queue preemption based on fairness.
- Customized resource types (like GPU) scheduling support.
- Rich placement constraints support.
- Automatically map incoming container requests to queues by policies.
- Node partition: partition cluster to sub-clusters with dedicated quota/ACL management.
Integration with K8s:
The k8shim provides the integration for K8s scheduling and supported features include:
- Predicates: All kinds of predicates such as node-selector, pod affinity/anti-affinity, taint/tolerant, etc.
- Persistent volumes: We have verified hostpath, EBS, NFS, etc.
- K8s namespace awareness: YuniKorn support hierarchical of queues, does it mean you need to give up K8s namespace? Answer is no, with simple config, YuniKorn is able to support automatically map K8s namespaces to YuniKorn queues. All K8s-namespace-related ResourceQuota, permissions will be still valid.
- Metrics: Prometheus, Grafana integration.
- Cluster AutoScaler: YuniKorn can nicely work with Cluster AutoScaler (https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) to drive cluster scales up and down.
- K8s Events: YuniKorn also integrated with K8s events, so lots of information can be retrieved by using
kubectl describe pod.
Performance testing
We love high-performance software, and we made tremendous efforts to make it to support large scale cluster and high-churning tasks. Here’s Performance Test Result
Deployment model
Yunikorn can be deployed as a K8s custom scheduler, and take over all POD scheduling. An enhancement is open to improve coexistence behaviour of the YuniKorn scheduler with other Kubernetes schedulers, like the default scheduler, in the cluster: Co-existing with other K8s schedulers.
Verified K8s versions
| K8s Version | Support? |
|---|---|
| 1.12.x (or earlier) | X |
| 1.13.x | √ |
| 1.14.x | √ |
| 1.15.x | √ |
| 1.16.x | To be verified |
| 1.17.x | To be verified |
Web UI
YuniKorn has builtin web UIs for queue hierarchies and apps. See below:
Roadmap
Want to learn more about future of YuniKorn? You can find more information about what are already supported and future plans in the Road Map.
How to use
The simplest way to run YuniKorn is to build a docker image and then deployed to Kubernetes with a yaml file, running as a customized scheduler. Then you can run workloads with this scheduler. See more instructions from the User Guide.
How can I get involved?
Apache YuniKorn (Incubating) community includes engineers from Alibaba, Apple, Cloudera, Linkedin, Microsoft, Nvidia, Tencent, Uber, etc. (sorted by alphabet). Want to join the community? We welcome any form of contributions, code, documentation or suggestions!
To get involved, please read following resources.
- Logging an issue or improvement use the Reporting an issue Guide.
- Before you contributing code or documentation to YuniKorn, please read our Developer Guide.
- When you are coding use the Coding Guidelines.
- Please read How to Contribute to understand the procedure and guidelines of making contributions.
- We have periodically community sync ups in multiple timezones and languages, please find Community Syncup to attend online sync ups.
Other Resources
Demo videos
- Subscribe to YuniKorn Youtube Channel to get notification about new demos!
- Running YuniKorn on Kubernetes - a 12 minutes Hello-world demo
- YuniKorn configuration hot-refresh introduction
- Yunikorn scheduling and volumes on Kubernetes
- Yunikorn placement rules for applications
Communication channels
- Mailing lists are:
- for people wanting to contribute to the project: dev@yunikorn.apache.org subscribe unsubscribe archives
- JIRA issue updates: issues@yunikorn.apache.org subscribe unsubscribe archives
- for review messages and patches in GitHub pull requests reviews@yunikorn.apache.org subscribe unsubscribe archives
- We use Slack as our collaboration system, you can join us by accessing this link.
Currently, we have following channels in the workspace:
#yunikorn-devand#yunikorn-user.
Blog posts
Apache based blogs:
3rd party blog posts: