ApacheCon 2016

Most datacenters today are statically partitioned into siloed clusters dedicated to run various datacenter-scale applications including web services, databases, and batch/stream processing. This static partitioning model limits overall cluster utilization, decreases flexibility and causes operational challenges. There is an increasing need to integrate Apache Hadoop big data applications with other datacenter services like Apache Spark or Apache Kafka, ideally co-locating the data with the services that need it.

In this talk Adam and Mohit will show off Apache Myriad (incubating), which integrates Apache YARN with Apache Mesos. Mesos enables efficient resource sharing and isolation across a variety of distributed applications. This allows Hadoop workloads to run alongside other datacenter applications while sharing the same cluster resources, and hence breaking silos. Myriad’s multi-tenancy strategy improves overall cluster utilization and operational efficiency, while allowing cluster operators to run multiple isolated YARN clusters on the same hardware.