Kubernetes Scheduler Internals: Filtering (Predicates), Scoring (Priorities), and Affinities

A batch analytics Pod configured with high resource requirements (requests: { memory: 32Gi, cpu: 16 }) is submitted to a 100-node cluster.

At the moment of submission:

• Node 01 has 64GB of total RAM, but 60GB is already allocated to active Pods (only 4GB available).
• Node 02 has 64GB of total RAM, and only 8GB is allocated (56GB available).
• Node 03 is marked with a taint: dedicated=gpu:NoSchedule.

If the cluster placed the 32GB analytics Pod onto Node 01, Node 01 would suffer immediate memory exhaustion and evict critical production web services.

Within 3 milliseconds, the kube-scheduler evaluates all 100 nodes, eliminates Node 01 and Node 03, selects Node 02, and writes nodeName: node-02 into the Pod’s specification in etcd.

How does the Kubernetes Scheduler make intelligent, sub-millisecond node placement decisions across thousands of cluster nodes?

The answer lies in the 2-Phase Scheduling Algorithm: Filtering (Predicates) and Scoring (Priorities).

---

1. The 2-Phase Scheduling Pipeline

The kube-scheduler runs as a control plane daemon watching for unassigned Pods (nodeName == "").

When an unassigned Pod is discovered, the Scheduler processes candidate nodes through a 2-phase pipeline:

[ Unassigned Pod Object ]
            |
            v
+-------------------------------------------------------------------+
| PHASE 1: FILTERING (Predicates)                                   |
| Eliminates unviable nodes that fail hard requirements             |
| (e.g., insufficient CPU/RAM, taints, missing volumes)             |
+-------------------------------------------------------------------+
            |
            v  List of Feasible Candidate Nodes
+-------------------------------------------------------------------+
| PHASE 2: SCORING (Priorities)                                     |
| Ranks surviving candidate nodes on a score scale (0 to 100)       |
| (e.g., resource balance, image locality, pod affinity)            |
+-------------------------------------------------------------------+
            |
            v  Node with Highest Total Score
+-------------------------------------------------------------------+
| BINDING PHASE                                                     |
| Writes `nodeName: node-02` to API Server via REST Bind Call       |
+-------------------------------------------------------------------+

---

2. Phase 1: Filtering (Predicates)

In the Filtering Phase, the Scheduler applies a series of hard predicate checks to filter out nodes that cannot run the Pod:

Core Filtering Predicates:

1. NodeResourcesFit: Checks if a node has sufficient unallocated CPU, RAM, and ephemeral storage to satisfy the Pod’s requests.
2. NodeName: Checks if the PodSpec explicitly requested a specific host via nodeName.
3. NodePorts: Checks if a requested NodePort is already bound by another container on the node.
4. PodFitsHostPorts: Checks if host ports requested by the container are already in use.
5. NodeUnschedulable: Filters out nodes marked as unschedulable (e.g., during kubectl drain).
6. NodeTaints: Checks if the node has Taints that the Pod lacks matching Tolerations for.

If zero nodes survive the Filtering Phase, the Pod remains in Pending status with event reason FailedScheduling.

---

3. Phase 2: Scoring (Priorities)

In the Scoring Phase, the Scheduler evaluates all surviving candidate nodes using weighted scoring plugins, assigning each node a score from 0 to 100:

$\text{Final Score} = \sum (\text{Plugin Score} \times \text{Plugin Weight})$

Core Scoring Plugins:

1. NodeResourcesBalancedAllocation: Prefers nodes that achieve a balanced ratio of CPU and RAM utilization after placing the Pod.
2. ImageLocality: Grants higher scores to nodes that already have the requested container image cached locally, eliminating network layer download delays!
3. NodeAffinity: Scores nodes based on soft preference match rules (preferredDuringSchedulingIgnoredDuringExecution).
4. PodTopologySpread: Grants higher scores to nodes that spread Pod replicas evenly across availability zones to maximize fault tolerance.

The node with the highest cumulative score is selected. (If a tie occurs, the Scheduler selects a node at random).

---

4. Advanced Placement: Affinities, Taints, and Tolerations

Kubernetes provides three key abstractions for controlling workload placement:

                          PLACEMENT MECHANISMS
                                    |
     +------------------------------+------------------------------+
     |                              |                              |
     v                              v                              v
[ Node Affinity ]           [ Pod Anti-Affinity ]       [ Taints & Tolerations ]
Attracts Pods to            Spreads Pod replicas        Repels Pods from nodes
specific node labels        across zones / nodes        unless Pod has Toleration

A. Node Affinity

Instructs the Scheduler to place Pods on nodes with specific labels (e.g., disktype=ssd or topology.kubernetes.io/zone=us-east-1a):

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution: # Hard Rule
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values: [ "ssd" ]

---

B. Pod Anti-Affinity (High Availability Spreading)

Prevents two replicas of the same critical service from running on the exact same worker node or availability zone:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values: [ "payment-api" ]
      topologyKey: "kubernetes.io/hostname"

---

C. Taints and Tolerations

While Affinities attract Pods to nodes, Taints allow a node to REPEL Pods.

# Apply a Taint to a node with GPU hardware:
kubectl taint nodes gpu-node-01 gpu=true:NoSchedule

Any standard Pod without a matching Toleration will be filtered out by the Scheduler during Phase 1:

# Pod Spec Toleration allowing placement on GPU node:
tolerations:
- key: "gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

---

Summary & Next Steps

The kube-scheduler optimizes cluster resource utilization and workload availability:

• Filtering (Predicates) eliminates unviable nodes based on CPU/RAM requests, port conflicts, and taints.
• Scoring (Priorities) ranks candidate nodes from 0 to 100 based on resource balance, image locality, and topology spreading.
• Node Affinity attracts Pods to nodes matching label expressions.
• Pod Anti-Affinity prevents single points of failure by spreading Pod replicas across nodes or availability zones.
• Taints & Tolerations allow nodes to repel workloads unless Pods carry explicit matching tolerations.

In the next article, we examine Cluster Security & Authorization: RBAC, ServiceAccounts, and Admission Webhooks.

References & Further Reading

1. AWS Open Source. Karpenter Just-in-Time Infrastructure Provisioning for Kubernetes. Karpenter Docs.
2. CNCF SIG Autoscaling. Kubernetes Cluster Autoscaler Architecture Design Doc. CNCF GitHub.
3. Gregg, B. (2020). Systems Performance: Enterprise and the Cloud (2nd Edition) — Capacity Planning. Addison-Wesley.

---