1 - Client Authentication (v1)

Resource Types

ExecCredential

ExecCredential is used by exec-based plugins to communicate credentials to HTTP transports.

FieldDescription
apiVersion
string
client.authentication.k8s.io/v1
kind
string
ExecCredential
spec [Required]
ExecCredentialSpec

Spec holds information passed to the plugin by the transport.

status
ExecCredentialStatus

Status is filled in by the plugin and holds the credentials that the transport should use to contact the API.

Cluster

Appears in:

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to.

To ensure that this struct contains everything someone would need to communicate with a kubernetes cluster (just like they would via a kubeconfig), the fields should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception of CertificateAuthority, since CA data will always be passed to the plugin as bytes.

FieldDescription
server [Required]
string

Server is the address of the kubernetes cluster (https://hostname:port).

tls-server-name
string

TLSServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname used to contact the server is used.

insecure-skip-tls-verify
bool

InsecureSkipTLSVerify skips the validity check for the server's certificate. This will make your HTTPS connections insecure.

certificate-authority-data
[]byte

CAData contains PEM-encoded certificate authority certificates. If empty, system roots should be used.

proxy-url
string

ProxyURL is the URL to the proxy to be used for all requests to this cluster.

config
k8s.io/apimachinery/pkg/runtime.RawExtension

Config holds additional config data that is specific to the exec plugin with regards to the cluster being authenticated to.

This data is sourced from the clientcmd Cluster object's extensions[client.authentication.k8s.io/exec] field:

clusters:

  • name: my-cluster cluster: ... extensions:
    • name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config extension: audience: 06e3fbd18de8 # arbitrary config

In some environments, the user config may be exactly the same across many clusters (i.e. call this exec plugin) minus some details that are specific to each cluster such as the audience. This field allows the per cluster config to be directly specified with the cluster info. Using this field to store secret data is not recommended as one of the prime benefits of exec plugins is that no secrets need to be stored directly in the kubeconfig.

ExecCredentialSpec

Appears in:

ExecCredentialSpec holds request and runtime specific information provided by the transport.

FieldDescription
cluster
Cluster

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to. Note that Cluster is non-nil only when provideClusterInfo is set to true in the exec provider config (i.e., ExecConfig.ProvideClusterInfo).

interactive [Required]
bool

Interactive declares whether stdin has been passed to this exec plugin.

ExecCredentialStatus

Appears in:

ExecCredentialStatus holds credentials for the transport to use.

Token and ClientKeyData are sensitive fields. This data should only be transmitted in-memory between client and exec plugin process. Exec plugin itself should at least be protected via file permissions.

FieldDescription
expirationTimestamp
meta/v1.Time

ExpirationTimestamp indicates a time when the provided credentials expire.

token [Required]
string

Token is a bearer token used by the client for request authentication.

clientCertificateData [Required]
string

PEM-encoded client TLS certificates (including intermediates, if any).

clientKeyData [Required]
string

PEM-encoded private key for the above certificate.

2 - Client Authentication (v1beta1)

Resource Types

ExecCredential

ExecCredential is used by exec-based plugins to communicate credentials to HTTP transports.

FieldDescription
apiVersion
string
client.authentication.k8s.io/v1beta1
kind
string
ExecCredential
spec [Required]
ExecCredentialSpec

Spec holds information passed to the plugin by the transport.

status
ExecCredentialStatus

Status is filled in by the plugin and holds the credentials that the transport should use to contact the API.

Cluster

Appears in:

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to.

To ensure that this struct contains everything someone would need to communicate with a kubernetes cluster (just like they would via a kubeconfig), the fields should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception of CertificateAuthority, since CA data will always be passed to the plugin as bytes.

FieldDescription
server [Required]
string

Server is the address of the kubernetes cluster (https://hostname:port).

tls-server-name
string

TLSServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname used to contact the server is used.

insecure-skip-tls-verify
bool

InsecureSkipTLSVerify skips the validity check for the server's certificate. This will make your HTTPS connections insecure.

certificate-authority-data
[]byte

CAData contains PEM-encoded certificate authority certificates. If empty, system roots should be used.

proxy-url
string

ProxyURL is the URL to the proxy to be used for all requests to this cluster.

config
k8s.io/apimachinery/pkg/runtime.RawExtension

Config holds additional config data that is specific to the exec plugin with regards to the cluster being authenticated to.

This data is sourced from the clientcmd Cluster object's extensions[client.authentication.k8s.io/exec] field:

clusters:

  • name: my-cluster cluster: ... extensions:
    • name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config extension: audience: 06e3fbd18de8 # arbitrary config

In some environments, the user config may be exactly the same across many clusters (i.e. call this exec plugin) minus some details that are specific to each cluster such as the audience. This field allows the per cluster config to be directly specified with the cluster info. Using this field to store secret data is not recommended as one of the prime benefits of exec plugins is that no secrets need to be stored directly in the kubeconfig.

ExecCredentialSpec

Appears in:

ExecCredentialSpec holds request and runtime specific information provided by the transport.

FieldDescription
cluster
Cluster

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to. Note that Cluster is non-nil only when provideClusterInfo is set to true in the exec provider config (i.e., ExecConfig.ProvideClusterInfo).

interactive [Required]
bool

Interactive declares whether stdin has been passed to this exec plugin.

ExecCredentialStatus

Appears in:

ExecCredentialStatus holds credentials for the transport to use.

Token and ClientKeyData are sensitive fields. This data should only be transmitted in-memory between client and exec plugin process. Exec plugin itself should at least be protected via file permissions.

FieldDescription
expirationTimestamp
meta/v1.Time

ExpirationTimestamp indicates a time when the provided credentials expire.

token [Required]
string

Token is a bearer token used by the client for request authentication.

clientCertificateData [Required]
string

PEM-encoded client TLS certificates (including intermediates, if any).

clientKeyData [Required]
string

PEM-encoded private key for the above certificate.

3 - Event Rate Limit Configuration (v1alpha1)

Resource Types

Configuration

Configuration provides configuration for the EventRateLimit admission controller.

FieldDescription
apiVersion
string
eventratelimit.admission.k8s.io/v1alpha1
kind
string
Configuration
limits [Required]
[]Limit

limits are the limits to place on event queries received. Limits can be placed on events received server-wide, per namespace, per user, and per source+object. At least one limit is required.

Limit

Appears in:

Limit is the configuration for a particular limit type

FieldDescription
type [Required]
LimitType

type is the type of limit to which this configuration applies

qps [Required]
int32

qps is the number of event queries per second that are allowed for this type of limit. The qps and burst fields are used together to determine if a particular event query is accepted. The qps determines how many queries are accepted once the burst amount of queries has been exhausted.

burst [Required]
int32

burst is the burst number of event queries that are allowed for this type of limit. The qps and burst fields are used together to determine if a particular event query is accepted. The burst determines the maximum size of the allowance granted for a particular bucket. For example, if the burst is 10 and the qps is 3, then the admission control will accept 10 queries before blocking any queries. Every second, 3 more queries will be allowed. If some of that allowance is not used, then it will roll over to the next second, until the maximum allowance of 10 is reached.

cacheSize
int32

cacheSize is the size of the LRU cache for this type of limit. If a bucket is evicted from the cache, then the allowance for that bucket is reset. If more queries are later received for an evicted bucket, then that bucket will re-enter the cache with a clean slate, giving that bucket a full allowance of burst queries.

The default cache size is 4096.

If limitType is 'server', then cacheSize is ignored.

LimitType

(Alias of string)

Appears in:

LimitType is the type of the limit (e.g., per-namespace)

4 - Image Policy API (v1alpha1)

Resource Types

ImageReview

ImageReview checks if the set of images in a pod are allowed.

FieldDescription
apiVersion
string
imagepolicy.k8s.io/v1alpha1
kind
string
ImageReview
metadata
meta/v1.ObjectMeta

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec [Required]
ImageReviewSpec

Spec holds information about the pod being evaluated

status
ImageReviewStatus

Status is filled in by the backend and indicates whether the pod should be allowed.

ImageReviewContainerSpec

Appears in:

ImageReviewContainerSpec is a description of a container within the pod creation request.

FieldDescription
image
string

This can be in the form image:tag or image@SHA:012345679abcdef.

ImageReviewSpec

Appears in:

ImageReviewSpec is a description of the pod creation request.

FieldDescription
containers
[]ImageReviewContainerSpec

Containers is a list of a subset of the information in each container of the Pod being created.

annotations
map[string]string

Annotations is a list of key-value pairs extracted from the Pod's annotations. It only includes keys which match the pattern *.image-policy.k8s.io/*. It is up to each webhook backend to determine how to interpret these annotations, if at all.

namespace
string

Namespace is the namespace the pod is being created in.

ImageReviewStatus

Appears in:

ImageReviewStatus is the result of the review for the pod creation request.

FieldDescription
allowed [Required]
bool

Allowed indicates that all images were allowed to be run.

reason
string

Reason should be empty unless Allowed is false in which case it may contain a short description of what is wrong. Kubernetes may truncate excessively long errors when displaying to the user.

auditAnnotations
map[string]string

AuditAnnotations will be added to the attributes object of the admission controller request using 'AddAnnotation'. The keys should be prefix-less (i.e., the admission controller will add an appropriate prefix).

5 - kube-apiserver Audit Configuration (v1)

Resource Types

Event

Appears in:

Event captures all the information that can be included in an API audit log.

FieldDescription
apiVersion
string
audit.k8s.io/v1
kind
string
Event
level [Required]
Level

AuditLevel at which event was generated

auditID [Required]
k8s.io/apimachinery/pkg/types.UID

Unique audit ID, generated for each request.

stage [Required]
Stage

Stage of the request handling when this event instance was generated.

requestURI [Required]
string

RequestURI is the request URI as sent by the client to a server.

verb [Required]
string

Verb is the kubernetes verb associated with the request. For non-resource requests, this is the lower-cased HTTP method.

user [Required]
authentication/v1.UserInfo

Authenticated user information.

impersonatedUser
authentication/v1.UserInfo

Impersonated user information.

sourceIPs
[]string

Source IPs, from where the request originated and intermediate proxies. The source IPs are listed from (in order):

  1. X-Forwarded-For request header IPs
  2. X-Real-Ip header, if not present in the X-Forwarded-For list
  3. The remote address for the connection, if it doesn't match the last IP in the list up to here (X-Forwarded-For or X-Real-Ip). Note: All but the last IP can be arbitrarily set by the client.
userAgent
string

UserAgent records the user agent string reported by the client. Note that the UserAgent is provided by the client, and must not be trusted.

objectRef
ObjectReference

Object reference this request is targeted at. Does not apply for List-type requests, or non-resource requests.

responseStatus
meta/v1.Status

The response status, populated even when the ResponseObject is not a Status type. For successful responses, this will only include the Code and StatusSuccess. For non-status type error responses, this will be auto-populated with the error Message.

requestObject
k8s.io/apimachinery/pkg/runtime.Unknown

API object from the request, in JSON format. The RequestObject is recorded as-is in the request (possibly re-encoded as JSON), prior to version conversion, defaulting, admission or merging. It is an external versioned object type, and may not be a valid object on its own. Omitted for non-resource requests. Only logged at Request Level and higher.

responseObject
k8s.io/apimachinery/pkg/runtime.Unknown

API object returned in the response, in JSON. The ResponseObject is recorded after conversion to the external type, and serialized as JSON. Omitted for non-resource requests. Only logged at Response Level.

requestReceivedTimestamp
meta/v1.MicroTime

Time the request reached the apiserver.

stageTimestamp
meta/v1.MicroTime

Time the request reached current audit stage.

annotations
map[string]string

Annotations is an unstructured key value map stored with an audit event that may be set by plugins invoked in the request serving chain, including authentication, authorization and admission plugins. Note that these annotations are for the audit event, and do not correspond to the metadata.annotations of the submitted object. Keys should uniquely identify the informing component to avoid name collisions (e.g. podsecuritypolicy.admission.k8s.io/policy). Values should be short. Annotations are included in the Metadata level.

EventList

EventList is a list of audit Events.

FieldDescription
apiVersion
string
audit.k8s.io/v1
kind
string
EventList
metadata
meta/v1.ListMeta
No description provided.
items [Required]
[]Event
No description provided.

Policy

Appears in:

Policy defines the configuration of audit logging, and the rules for how different request categories are logged.

FieldDescription
apiVersion
string
audit.k8s.io/v1
kind
string
Policy
metadata
meta/v1.ObjectMeta

ObjectMeta is included for interoperability with API infrastructure.

Refer to the Kubernetes API documentation for the fields of the metadata field.
rules [Required]
[]PolicyRule

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

omitStages
[]Stage

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

omitManagedFields
bool

OmitManagedFields indicates whether to omit the managed fields of the request and response bodies from being written to the API audit log. This is used as a global default - a value of 'true' will omit the managed fileds, otherwise the managed fields will be included in the API audit log. Note that this can also be specified per rule in which case the value specified in a rule will override the global default.

PolicyList

PolicyList is a list of audit Policies.

FieldDescription
apiVersion
string
audit.k8s.io/v1
kind
string
PolicyList
metadata
meta/v1.ListMeta
No description provided.
items [Required]
[]Policy
No description provided.

GroupResources

Appears in:

GroupResources represents resource kinds in an API group.

FieldDescription
group
string

Group is the name of the API group that contains the resources. The empty string represents the core API group.

resources
[]string

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

resourceNames
[]string

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

Level

(Alias of string)

Appears in:

Level defines the amount of information logged during auditing

ObjectReference

Appears in:

ObjectReference contains enough information to let you inspect or modify the referred object.

FieldDescription
resource
string
No description provided.
namespace
string
No description provided.
name
string
No description provided.
uid
k8s.io/apimachinery/pkg/types.UID
No description provided.
apiGroup
string

APIGroup is the name of the API group that contains the referred object. The empty string represents the core API group.

apiVersion
string

APIVersion is the version of the API group that contains the referred object.

resourceVersion
string
No description provided.
subresource
string
No description provided.

PolicyRule

Appears in:

PolicyRule maps requests based off metadata to an audit Level. Requests must match the rules of every field (an intersection of rules).

FieldDescription
level [Required]
Level

The Level that requests matching this rule are recorded at.

users
[]string

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups
[]string

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs
[]string

The verbs that match this rule. An empty list implies every verb.

resources
[]GroupResources

Resources that this rule matches. An empty list implies all kinds in all API groups.

namespaces
[]string

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs
[]string

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages
[]Stage

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

omitManagedFields
bool

OmitManagedFields indicates whether to omit the managed fields of the request and response bodies from being written to the API audit log.

  • a value of 'true' will drop the managed fields from the API audit log
  • a value of 'false' indicates that the managed fileds should be included in the API audit log Note that the value, if specified, in this rule will override the global default If a value is not specified then the global default specified in Policy.OmitManagedFields will stand.

Stage

(Alias of string)

Appears in:

Stage defines the stages in request handling that audit events may be generated.

6 - kube-apiserver Configuration (v1)

Package v1 is the v1 version of the API.

Resource Types

AdmissionConfiguration

AdmissionConfiguration provides versioned configuration for admission controllers.

FieldDescription
apiVersion
string
apiserver.config.k8s.io/v1
kind
string
AdmissionConfiguration
plugins
[]AdmissionPluginConfiguration

Plugins allows specifying a configuration per admission control plugin.

AdmissionPluginConfiguration

Appears in:

AdmissionPluginConfiguration provides the configuration for a single plug-in.

FieldDescription
name [Required]
string

Name is the name of the admission controller. It must match the registered admission plugin name.

path
string

Path is the path to a configuration file that contains the plugin's configuration

configuration
k8s.io/apimachinery/pkg/runtime.Unknown

Configuration is an embedded configuration object to be used as the plugin's configuration. If present, it will be used instead of the path to the configuration file.

7 - kube-apiserver Configuration (v1alpha1)

Package v1alpha1 is the v1alpha1 version of the API.

Resource Types

AdmissionConfiguration

AdmissionConfiguration provides versioned configuration for admission controllers.

FieldDescription
apiVersion
string
apiserver.k8s.io/v1alpha1
kind
string
AdmissionConfiguration
plugins
[]AdmissionPluginConfiguration

Plugins allows specifying a configuration per admission control plugin.

EgressSelectorConfiguration

EgressSelectorConfiguration provides versioned configuration for egress selector clients.

FieldDescription
apiVersion
string
apiserver.k8s.io/v1alpha1
kind
string
EgressSelectorConfiguration
egressSelections [Required]
[]EgressSelection

connectionServices contains a list of egress selection client configurations

TracingConfiguration

TracingConfiguration provides versioned configuration for tracing clients.

FieldDescription
apiVersion
string
apiserver.k8s.io/v1alpha1
kind
string
TracingConfiguration
endpoint
string

Endpoint of the collector that's running on the control-plane node. The APIServer uses the egressType ControlPlane when sending data to the collector. The syntax is defined in https://github.com/grpc/grpc/blob/master/doc/naming.md. Defaults to the otlpgrpc default, localhost:4317 The connection is insecure, and does not support TLS.

samplingRatePerMillion
int32

SamplingRatePerMillion is the number of samples to collect per million spans. Defaults to 0.

AdmissionPluginConfiguration

Appears in:

AdmissionPluginConfiguration provides the configuration for a single plug-in.

FieldDescription
name [Required]
string

Name is the name of the admission controller. It must match the registered admission plugin name.

path
string

Path is the path to a configuration file that contains the plugin's configuration

configuration
k8s.io/apimachinery/pkg/runtime.Unknown

Configuration is an embedded configuration object to be used as the plugin's configuration. If present, it will be used instead of the path to the configuration file.

Connection

Appears in:

Connection provides the configuration for a single egress selection client.

FieldDescription
proxyProtocol [Required]
ProtocolType

Protocol is the protocol used to connect from client to the konnectivity server.

transport
Transport

Transport defines the transport configurations we use to dial to the konnectivity server. This is required if ProxyProtocol is HTTPConnect or GRPC.

EgressSelection

Appears in:

EgressSelection provides the configuration for a single egress selection client.

FieldDescription
name [Required]
string

name is the name of the egress selection. Currently supported values are "controlplane", "master", "etcd" and "cluster" The "master" egress selector is deprecated in favor of "controlplane"

connection [Required]
Connection

connection is the exact information used to configure the egress selection

ProtocolType

(Alias of string)

Appears in:

ProtocolType is a set of valid values for Connection.ProtocolType

TCPTransport

Appears in:

TCPTransport provides the information to connect to konnectivity server via TCP

FieldDescription
url [Required]
string

URL is the location of the konnectivity server to connect to. As an example it might be "https://127.0.0.1:8131"

tlsConfig
TLSConfig

TLSConfig is the config needed to use TLS when connecting to konnectivity server

TLSConfig

Appears in:

TLSConfig provides the authentication information to connect to konnectivity server Only used with TCPTransport

FieldDescription
caBundle
string

caBundle is the file location of the CA to be used to determine trust with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.

clientKey
string

clientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// Must be configured if TCPTransport.URL is prefixed with https://

clientCert
string

clientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// Must be configured if TCPTransport.URL is prefixed with https://

Transport

Appears in:

Transport defines the transport configurations we use to dial to the konnectivity server

FieldDescription
tcp
TCPTransport

TCP is the TCP configuration for communicating with the konnectivity server via TCP ProxyProtocol of GRPC is not supported with TCP transport at the moment Requires at least one of TCP or UDS to be set

uds
UDSTransport

UDS is the UDS configuration for communicating with the konnectivity server via UDS Requires at least one of TCP or UDS to be set

UDSTransport

Appears in:

UDSTransport provides the information to connect to konnectivity server via UDS

FieldDescription
udsName [Required]
string

UDSName is the name of the unix domain socket to connect to konnectivity server This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)

8 - kube-apiserver Encryption Configuration (v1)

Package v1 is the v1 version of the API.

Resource Types

EncryptionConfiguration

EncryptionConfiguration stores the complete configuration for encryption providers.

FieldDescription
apiVersion
string
apiserver.config.k8s.io/v1
kind
string
EncryptionConfiguration
resources [Required]
[]ResourceConfiguration

resources is a list containing resources, and their corresponding encryption providers.

AESConfiguration

Appears in:

AESConfiguration contains the API configuration for an AES transformer.

FieldDescription
keys [Required]
[]Key

keys is a list of keys to be used for creating the AES transformer. Each key has to be 32 bytes long for AES-CBC and 16, 24 or 32 bytes for AES-GCM.

IdentityConfiguration

Appears in:

IdentityConfiguration is an empty struct to allow identity transformer in provider configuration.

KMSConfiguration

Appears in:

KMSConfiguration contains the name, cache size and path to configuration file for a KMS based envelope transformer.

FieldDescription
name [Required]
string

name is the name of the KMS plugin to be used.

cachesize
int32

cachesize is the maximum number of secrets which are cached in memory. The default value is 1000. Set to a negative value to disable caching.

endpoint [Required]
string

endpoint is the gRPC server listening address, for example "unix:///var/run/kms-provider.sock".

timeout
meta/v1.Duration

timeout for gRPC calls to kms-plugin (ex. 5s). The default is 3 seconds.

Key

Appears in:

Key contains name and secret of the provided key for a transformer.

FieldDescription
name [Required]
string

name is the name of the key to be used while storing data to disk.

secret [Required]
string

secret is the actual key, encoded in base64.

ProviderConfiguration

Appears in:

ProviderConfiguration stores the provided configuration for an encryption provider.

FieldDescription
aesgcm [Required]
AESConfiguration

aesgcm is the configuration for the AES-GCM transformer.

aescbc [Required]
AESConfiguration

aescbc is the configuration for the AES-CBC transformer.

secretbox [Required]
SecretboxConfiguration

secretbox is the configuration for the Secretbox based transformer.

identity [Required]
IdentityConfiguration

identity is the (empty) configuration for the identity transformer.

kms [Required]
KMSConfiguration

kms contains the name, cache size and path to configuration file for a KMS based envelope transformer.

ResourceConfiguration

Appears in:

ResourceConfiguration stores per resource configuration.

FieldDescription
resources [Required]
[]string

resources is a list of kubernetes resources which have to be encrypted.

providers [Required]
[]ProviderConfiguration

providers is a list of transformers to be used for reading and writing the resources to disk. eg: aesgcm, aescbc, secretbox, identity.

SecretboxConfiguration

Appears in:

SecretboxConfiguration contains the API configuration for an Secretbox transformer.

FieldDescription
keys [Required]
[]Key

keys is a list of keys to be used for creating the Secretbox transformer. Each key has to be 32 bytes long.

9 - kube-proxy Configuration (v1alpha1)

Resource Types

KubeProxyConfiguration

KubeProxyConfiguration contains everything necessary to configure the Kubernetes proxy server.

FieldDescription
apiVersion
string
kubeproxy.config.k8s.io/v1alpha1
kind
string
KubeProxyConfiguration
featureGates [Required]
map[string]bool

featureGates is a map of feature names to bools that enable or disable alpha/experimental features.

bindAddress [Required]
string

bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0 for all interfaces)

healthzBindAddress [Required]
string

healthzBindAddress is the IP address and port for the health check server to serve on, defaulting to 0.0.0.0:10256

metricsBindAddress [Required]
string

metricsBindAddress is the IP address and port for the metrics server to serve on, defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)

bindAddressHardFail [Required]
bool

bindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit

enableProfiling [Required]
bool

enableProfiling enables profiling via web interface on /debug/pprof handler. Profiling handlers will be handled by metrics server.

clusterCIDR [Required]
string

clusterCIDR is the CIDR range of the pods in the cluster. It is used to bridge traffic coming from outside of the cluster. If not provided, no off-cluster bridging will be performed.

hostnameOverride [Required]
string

hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.

clientConnection [Required]
ClientConnectionConfiguration

clientConnection specifies the kubeconfig file and client connection settings for the proxy server to use when communicating with the apiserver.

iptables [Required]
KubeProxyIPTablesConfiguration

iptables contains iptables-related configuration options.

ipvs [Required]
KubeProxyIPVSConfiguration

ipvs contains ipvs-related configuration options.

oomScoreAdj [Required]
int32

oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]

mode [Required]
ProxyMode

mode specifies which proxy mode to use.

portRange [Required]
string

portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.

udpIdleTimeout [Required]
meta/v1.Duration

udpIdleTimeout is how long an idle UDP connection will be kept open (e.g. '250ms', '2s'). Must be greater than 0. Only applicable for proxyMode=userspace.

conntrack [Required]
KubeProxyConntrackConfiguration

conntrack contains conntrack-related configuration options.

configSyncPeriod [Required]
meta/v1.Duration

configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater than 0.

nodePortAddresses [Required]
[]string

nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid IP blocks. These values are as a parameter to select the interfaces where nodeport works. In case someone would like to expose a service on localhost for local visit and some other interfaces for particular purpose, a list of IP blocks would do that. If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort. If set it to a non-zero IP block, kube-proxy will filter that down to just the IPs that applied to the node. An empty string slice is meant to select all network interfaces.

winkernel [Required]
KubeProxyWinkernelConfiguration

winkernel contains winkernel-related configuration options.

showHiddenMetricsForVersion [Required]
string

ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.

detectLocalMode [Required]
LocalMode

DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR

detectLocal [Required]
DetectLocalConfiguration

DetectLocal contains optional configuration settings related to DetectLocalMode.

DetectLocalConfiguration

Appears in:

DetectLocalConfiguration contains optional settings related to DetectLocalMode option

FieldDescription
bridgeInterface [Required]
string

BridgeInterface is a string argument which represents a single bridge interface name. Kube-proxy considers traffic as local if originating from this given bridge. This argument should be set if DetectLocalMode is set to LocalModeBridgeInterface.

interfaceNamePrefix [Required]
string

InterfaceNamePrefix is a string argument which represents a single interface prefix name. Kube-proxy considers traffic as local if originating from one or more interfaces which match the given prefix. This argument should be set if DetectLocalMode is set to LocalModeInterfaceNamePrefix.

KubeProxyConntrackConfiguration

Appears in:

KubeProxyConntrackConfiguration contains conntrack settings for the Kubernetes proxy server.

FieldDescription
maxPerCore [Required]
int32

maxPerCore is the maximum number of NAT connections to track per CPU core (0 to leave the limit as-is and ignore min).

min [Required]
int32

min is the minimum value of connect-tracking records to allocate, regardless of conntrackMaxPerCore (set maxPerCore=0 to leave the limit as-is).

tcpEstablishedTimeout [Required]
meta/v1.Duration

tcpEstablishedTimeout is how long an idle TCP connection will be kept open (e.g. '2s'). Must be greater than 0 to set.

tcpCloseWaitTimeout [Required]
meta/v1.Duration

tcpCloseWaitTimeout is how long an idle conntrack entry in CLOSE_WAIT state will remain in the conntrack table. (e.g. '60s'). Must be greater than 0 to set.

KubeProxyIPTablesConfiguration

Appears in:

KubeProxyIPTablesConfiguration contains iptables-related configuration details for the Kubernetes proxy server.

FieldDescription
masqueradeBit [Required]
int32

masqueradeBit is the bit of the iptables fwmark space to use for SNAT if using the pure iptables proxy mode. Values must be within the range [0, 31].

masqueradeAll [Required]
bool

masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode.

syncPeriod [Required]
meta/v1.Duration

syncPeriod is the period that iptables rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.

minSyncPeriod [Required]
meta/v1.Duration

minSyncPeriod is the minimum period that iptables rules are refreshed (e.g. '5s', '1m', '2h22m').

KubeProxyIPVSConfiguration

Appears in:

KubeProxyIPVSConfiguration contains ipvs-related configuration details for the Kubernetes proxy server.

FieldDescription
syncPeriod [Required]
meta/v1.Duration

syncPeriod is the period that ipvs rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.

minSyncPeriod [Required]
meta/v1.Duration

minSyncPeriod is the minimum period that ipvs rules are refreshed (e.g. '5s', '1m', '2h22m').

scheduler [Required]
string

ipvs scheduler

excludeCIDRs [Required]
[]string

excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch when cleaning up ipvs services.

strictARP [Required]
bool

strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface

tcpTimeout [Required]
meta/v1.Duration

tcpTimeout is the timeout value used for idle IPVS TCP sessions. The default value is 0, which preserves the current timeout value on the system.

tcpFinTimeout [Required]
meta/v1.Duration

tcpFinTimeout is the timeout value used for IPVS TCP sessions after receiving a FIN. The default value is 0, which preserves the current timeout value on the system.

udpTimeout [Required]
meta/v1.Duration

udpTimeout is the timeout value used for IPVS UDP packets. The default value is 0, which preserves the current timeout value on the system.

KubeProxyWinkernelConfiguration

Appears in:

KubeProxyWinkernelConfiguration contains Windows/HNS settings for the Kubernetes proxy server.

FieldDescription
networkName [Required]
string

networkName is the name of the network kube-proxy will use to create endpoints and policies

sourceVip [Required]
string

sourceVip is the IP address of the source VIP endoint used for NAT when loadbalancing

enableDSR [Required]
bool

enableDSR tells kube-proxy whether HNS policies should be created with DSR

rootHnsEndpointName [Required]
string

RootHnsEndpointName is the name of hnsendpoint that is attached to l2bridge for root network namespace

forwardHealthCheckVip [Required]
bool

ForwardHealthCheckVip forwards service VIP for health check port on Windows

LocalMode

(Alias of string)

Appears in:

LocalMode represents modes to detect local traffic from the node

ProxyMode

(Alias of string)

Appears in:

ProxyMode represents modes used by the Kubernetes proxy server.

Currently, three modes of proxy are available in Linux platform: 'userspace' (older, going to be EOL), 'iptables' (newer, faster), 'ipvs'(newest, better in performance and scalability).

Two modes of proxy are available in Windows platform: 'userspace'(older, stable) and 'kernelspace' (newer, faster).

In Linux platform, if proxy mode is blank, use the best-available proxy (currently iptables, but may change in the future). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to 'ipvs', and the fall back path is firstly iptables and then userspace.

In Windows platform, if proxy mode is blank, use the best-available proxy (currently userspace, but may change in the future). If winkernel proxy is selected, regardless of how, but the Windows kernel can't support this mode of proxy, this always falls back to the userspace proxy.

ClientConnectionConfiguration

Appears in:

ClientConnectionConfiguration contains details for constructing a client.

FieldDescription
kubeconfig [Required]
string

kubeconfig is the path to a KubeConfig file.

acceptContentTypes [Required]
string

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular client.

contentType [Required]
string

contentType is the content type used when sending data to the server from this client.

qps [Required]
float32

qps controls the number of queries per second allowed for this connection.

burst [Required]
int32

burst allows extra queries to accumulate when a client is exceeding its rate.

DebuggingConfiguration

Appears in:

DebuggingConfiguration holds configuration for Debugging related features.

FieldDescription
enableProfiling [Required]
bool

enableProfiling enables profiling via web interface host:port/debug/pprof/

enableContentionProfiling [Required]
bool

enableContentionProfiling enables lock contention profiling, if enableProfiling is true.

FormatOptions

Appears in:

FormatOptions contains options for the different logging formats.

FieldDescription
json [Required]
JSONOptions

[Experimental] JSON contains options for logging format "json".

JSONOptions

Appears in:

JSONOptions contains options for logging format "json".

FieldDescription
splitStream [Required]
bool

[Experimental] SplitStream redirects error messages to stderr while info messages go to stdout, with buffering. The default is to write both to stdout, without buffering.

infoBufferSize [Required]
k8s.io/apimachinery/pkg/api/resource.QuantityValue

[Experimental] InfoBufferSize sets the size of the info stream when using split streams. The default is zero, which disables buffering.

LeaderElectionConfiguration

Appears in:

LeaderElectionConfiguration defines the configuration of leader election clients for components that can run with leader election enabled.

FieldDescription
leaderElect [Required]
bool

leaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated components for high availability.

leaseDuration [Required]
meta/v1.Duration

leaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

renewDeadline [Required]
meta/v1.Duration

renewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

retryPeriod [Required]
meta/v1.Duration

retryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

resourceLock [Required]
string

resourceLock indicates the resource object type that will be used to lock during leader election cycles.

resourceName [Required]
string

resourceName indicates the name of resource object that will be used to lock during leader election cycles.

resourceNamespace [Required]
string

resourceName indicates the namespace of resource object that will be used to lock during leader election cycles.

LoggingConfiguration

Appears in:

LoggingConfiguration contains logging options Refer Logs Options for more information.

FieldDescription
format [Required]
string

Format Flag specifies the structure of log messages. default value of format is text

flushFrequency [Required]
time.Duration

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log flushes. Ignored if the selected logging backend writes log messages without buffering.

verbosity [Required]
uint32

Verbosity is the threshold that determines which log messages are logged. Default is zero which logs only the most important messages. Higher values enable additional messages. Error messages are always logged.

vmodule [Required]
VModuleConfiguration

VModule overrides the verbosity threshold for individual files. Only supported for "text" log format.

options [Required]
FormatOptions

[Experimental] Options holds additional parameters that are specific to the different logging formats. Only the options for the selected format get used, but all of them get validated.

VModuleConfiguration

(Alias of []k8s.io/component-base/config/v1alpha1.VModuleItem)

Appears in:

VModuleConfiguration is a collection of individual file names or patterns and the corresponding verbosity threshold.

10 - kube-scheduler Configuration (v1beta2)

Resource Types

ClientConnectionConfiguration

Appears in:

ClientConnectionConfiguration contains details for constructing a client.

FieldDescription
kubeconfig [Required]
string

kubeconfig is the path to a KubeConfig file.

acceptContentTypes [Required]
string

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular client.

contentType [Required]
string

contentType is the content type used when sending data to the server from this client.

qps [Required]
float32

qps controls the number of queries per second allowed for this connection.

burst [Required]
int32

burst allows extra queries to accumulate when a client is exceeding its rate.

DebuggingConfiguration

Appears in:

DebuggingConfiguration holds configuration for Debugging related features.

FieldDescription
enableProfiling [Required]
bool

enableProfiling enables profiling via web interface host:port/debug/pprof/

enableContentionProfiling [Required]
bool

enableContentionProfiling enables lock contention profiling, if enableProfiling is true.

FormatOptions

Appears in:

FormatOptions contains options for the different logging formats.

FieldDescription
json [Required]
JSONOptions

[Experimental] JSON contains options for logging format "json".

JSONOptions

Appears in:

JSONOptions contains options for logging format "json".

FieldDescription
splitStream [Required]
bool

[Experimental] SplitStream redirects error messages to stderr while info messages go to stdout, with buffering. The default is to write both to stdout, without buffering.

infoBufferSize [Required]
k8s.io/apimachinery/pkg/api/resource.QuantityValue

[Experimental] InfoBufferSize sets the size of the info stream when using split streams. The default is zero, which disables buffering.

LeaderElectionConfiguration

Appears in:

LeaderElectionConfiguration defines the configuration of leader election clients for components that can run with leader election enabled.

FieldDescription
leaderElect [Required]
bool

leaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated components for high availability.

leaseDuration [Required]
meta/v1.Duration

leaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

renewDeadline [Required]
meta/v1.Duration

renewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

retryPeriod [Required]
meta/v1.Duration

retryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

resourceLock [Required]
string

resourceLock indicates the resource object type that will be used to lock during leader election cycles.

resourceName [Required]
string

resourceName indicates the name of resource object that will be used to lock during leader election cycles.

resourceNamespace [Required]
string

resourceName indicates the namespace of resource object that will be used to lock during leader election cycles.

LoggingConfiguration

Appears in:

LoggingConfiguration contains logging options Refer Logs Options for more information.

FieldDescription
format [Required]
string

Format Flag specifies the structure of log messages. default value of format is text

flushFrequency [Required]
time.Duration

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log flushes. Ignored if the selected logging backend writes log messages without buffering.

verbosity [Required]
uint32

Verbosity is the threshold that determines which log messages are logged. Default is zero which logs only the most important messages. Higher values enable additional messages. Error messages are always logged.

vmodule [Required]
VModuleConfiguration

VModule overrides the verbosity threshold for individual files. Only supported for "text" log format.

options [Required]
FormatOptions

[Experimental] Options holds additional parameters that are specific to the different logging formats. Only the options for the selected format get used, but all of them get validated.

VModuleConfiguration

(Alias of []k8s.io/component-base/config/v1alpha1.VModuleItem)

Appears in:

VModuleConfiguration is a collection of individual file names or patterns and the corresponding verbosity threshold.

DefaultPreemptionArgs

DefaultPreemptionArgs holds arguments used to configure the DefaultPreemption plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
DefaultPreemptionArgs
minCandidateNodesPercentage [Required]
int32

MinCandidateNodesPercentage is the minimum number of candidates to shortlist when dry running preemption as a percentage of number of nodes. Must be in the range [0, 100]. Defaults to 10% of the cluster size if unspecified.

minCandidateNodesAbsolute [Required]
int32

MinCandidateNodesAbsolute is the absolute minimum number of candidates to shortlist. The likely number of candidates enumerated for dry running preemption is given by the formula: numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute) We say "likely" because there are other factors such as PDB violations that play a role in the number of candidates shortlisted. Must be at least 0 nodes. Defaults to 100 nodes if unspecified.

InterPodAffinityArgs

InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
InterPodAffinityArgs
hardPodAffinityWeight [Required]
int32

HardPodAffinityWeight is the scoring weight for existing pods with a matching hard affinity to the incoming pod.

KubeSchedulerConfiguration

KubeSchedulerConfiguration configures a scheduler

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
KubeSchedulerConfiguration
parallelism [Required]
int32

Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16

leaderElection [Required]
LeaderElectionConfiguration

LeaderElection defines the configuration of leader election client.

clientConnection [Required]
ClientConnectionConfiguration

ClientConnection specifies the kubeconfig file and client connection settings for the proxy server to use when communicating with the apiserver.

healthzBindAddress [Required]
string

Note: Both HealthzBindAddress and MetricsBindAddress fields are deprecated. Only empty address or port 0 is allowed. Anything else will fail validation. HealthzBindAddress is the IP address and port for the health check server to serve on.

metricsBindAddress [Required]
string

MetricsBindAddress is the IP address and port for the metrics server to serve on.

DebuggingConfiguration [Required]
DebuggingConfiguration
(Members of DebuggingConfiguration are embedded into this type.)

DebuggingConfiguration holds configuration for Debugging related features TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration

percentageOfNodesToScore [Required]
int32

PercentageOfNodesToScore is the percentage of all nodes that once found feasible for running a pod, the scheduler stops its search for more feasible nodes in the cluster. This helps improve scheduler's performance. Scheduler always tries to find at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. Example: if the cluster size is 500 nodes and the value of this flag is 30, then scheduler stops finding further feasible nodes once it finds 150 feasible ones. When the value is 0, default percentage (5%--50% based on the size of the cluster) of the nodes will be scored.

podInitialBackoffSeconds [Required]
int64

PodInitialBackoffSeconds is the initial backoff for unschedulable pods. If specified, it must be greater than 0. If this value is null, the default value (1s) will be used.

podMaxBackoffSeconds [Required]
int64

PodMaxBackoffSeconds is the max backoff for unschedulable pods. If specified, it must be greater than podInitialBackoffSeconds. If this value is null, the default value (10s) will be used.

profiles [Required]
[]KubeSchedulerProfile

Profiles are scheduling profiles that kube-scheduler supports. Pods can choose to be scheduled under a particular profile by setting its associated scheduler name. Pods that don't specify any scheduler name are scheduled with the "default-scheduler" profile, if present here.

extenders [Required]
[]Extender

Extenders are the list of scheduler extenders, each holding the values of how to communicate with the extender. These extenders are shared by all scheduler profiles.

NodeAffinityArgs

NodeAffinityArgs holds arguments to configure the NodeAffinity plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
NodeAffinityArgs
addedAffinity
core/v1.NodeAffinity

AddedAffinity is applied to all Pods additionally to the NodeAffinity specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes match). When AddedAffinity is used, some Pods with affinity requirements that match a specific Node (such as Daemonset Pods) might remain unschedulable.

NodeResourcesBalancedAllocationArgs

NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
NodeResourcesBalancedAllocationArgs
resources [Required]
[]ResourceSpec

Resources to be managed, the default is "cpu" and "memory" if not specified.

NodeResourcesFitArgs

NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
NodeResourcesFitArgs
ignoredResources [Required]
[]string

IgnoredResources is the list of resources that NodeResources fit filter should ignore. This doesn't apply to scoring.

ignoredResourceGroups [Required]
[]string

IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. e.g. if group is ["example.com"], it will ignore all resource names that begin with "example.com", such as "example.com/aaa" and "example.com/bbb". A resource group name can't contain '/'. This doesn't apply to scoring.

scoringStrategy [Required]
ScoringStrategy

ScoringStrategy selects the node resource scoring strategy. The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.

PodTopologySpreadArgs

PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
PodTopologySpreadArgs
defaultConstraints
[]core/v1.TopologySpreadConstraint

DefaultConstraints defines topology spread constraints to be applied to Pods that don't define any in pod.spec.topologySpreadConstraints. .defaultConstraints[*].labelSelectors must be empty, as they are deduced from the Pod's membership to Services, ReplicationControllers, ReplicaSets or StatefulSets. When not empty, .defaultingType must be "List".

defaultingType
PodTopologySpreadConstraintsDefaulting

DefaultingType determines how .defaultConstraints are deduced. Can be one of "System" or "List".

  • "System": Use kubernetes defined constraints that spread Pods among Nodes and Zones.
  • "List": Use constraints defined in .defaultConstraints.

Defaults to "System".

VolumeBindingArgs

VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta2
kind
string
VolumeBindingArgs
bindTimeoutSeconds [Required]
int64

BindTimeoutSeconds is the timeout in seconds in volume binding operation. Value must be non-negative integer. The value zero indicates no waiting. If this value is nil, the default value (600) will be used.

shape
[]UtilizationShapePoint

Shape specifies the points defining the score function shape, which is used to score nodes based on the utilization of statically provisioned PVs. The utilization is calculated by dividing the total requested storage of the pod by the total capacity of feasible PVs on each node. Each point contains utilization (ranges from 0 to 100) and its associated score (ranges from 0 to 10). You can turn the priority by specifying different scores for different utilization numbers. The default shape points are:

  1. 0 for 0 utilization
  2. 10 for 100 utilization All points must be sorted in increasing order by utilization.

Extender

Appears in:

Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, it is assumed that the extender chose not to provide that extension.

FieldDescription
urlPrefix [Required]
string

URLPrefix at which the extender is available

filterVerb [Required]
string

Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.

preemptVerb [Required]
string

Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.

prioritizeVerb [Required]
string

Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.

weight [Required]
int64

The numeric multiplier for the node scores that the prioritize call generates. The weight should be a positive integer

bindVerb [Required]
string

Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender can implement this function.

enableHTTPS [Required]
bool

EnableHTTPS specifies whether https should be used to communicate with the extender

tlsConfig [Required]
ExtenderTLSConfig

TLSConfig specifies the transport layer security config

httpTimeout [Required]
meta/v1.Duration

HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize timeout is ignored, k8s/other extenders priorities are used to select the node.

nodeCacheCapable [Required]
bool

NodeCacheCapable specifies that the extender is capable of caching node information, so the scheduler should only send minimal information about the eligible nodes assuming that the extender already cached full details of all nodes in the cluster

managedResources
[]ExtenderManagedResource

ManagedResources is a list of extended resources that are managed by this extender.

  • A pod will be sent to the extender on the Filter, Prioritize and Bind (if the extender is the binder) phases iff the pod requests at least one of the extended resources in this list. If empty or unspecified, all pods will be sent to this extender.
  • If IgnoredByScheduler is set to true for a resource, kube-scheduler will skip checking the resource in predicates.
ignorable [Required]
bool

Ignorable specifies if the extender is ignorable, i.e. scheduling should not fail when the extender returns an error or is not reachable.

ExtenderManagedResource

Appears in:

ExtenderManagedResource describes the arguments of extended resources managed by an extender.

FieldDescription
name [Required]
string

Name is the extended resource name.

ignoredByScheduler [Required]
bool

IgnoredByScheduler indicates whether kube-scheduler should ignore this resource when applying predicates.

ExtenderTLSConfig

Appears in:

ExtenderTLSConfig contains settings to enable TLS with extender

FieldDescription
insecure [Required]
bool

Server should be accessed without verifying the TLS certificate. For testing only.

serverName [Required]
string

ServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname used to contact the server is used.

certFile [Required]
string

Server requires TLS client certificate authentication

keyFile [Required]
string

Server requires TLS client certificate authentication

caFile [Required]
string

Trusted root certificates for server

certData [Required]
[]byte

CertData holds PEM-encoded bytes (typically read from a client certificate file). CertData takes precedence over CertFile

keyData [Required]
[]byte

KeyData holds PEM-encoded bytes (typically read from a client certificate key file). KeyData takes precedence over KeyFile

caData [Required]
[]byte

CAData holds PEM-encoded bytes (typically read from a root certificates bundle). CAData takes precedence over CAFile

KubeSchedulerProfile

Appears in:

KubeSchedulerProfile is a scheduling profile.

FieldDescription
schedulerName [Required]
string

SchedulerName is the name of the scheduler associated to this profile. If SchedulerName matches with the pod's "spec.schedulerName", then the pod is scheduled with this profile.

plugins [Required]
Plugins

Plugins specify the set of plugins that should be enabled or disabled. Enabled plugins are the ones that should be enabled in addition to the default plugins. Disabled plugins are any of the default plugins that should be disabled. When no enabled or disabled plugin is specified for an extension point, default plugins for that extension point will be used if there is any. If a QueueSort plugin is specified, the same QueueSort Plugin and PluginConfig must be specified for all profiles.

pluginConfig [Required]
[]PluginConfig

PluginConfig is an optional set of custom plugin arguments for each plugin. Omitting config args for a plugin is equivalent to using the default config for that plugin.

Plugin

Appears in:

Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.

FieldDescription
name [Required]
string

Name defines the name of plugin

weight [Required]
int32

Weight defines the weight of plugin, only used for Score plugins.

PluginConfig

Appears in:

PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. It is up to the plugin to process these Args.

FieldDescription
name [Required]
string

Name defines the name of plugin being configured

args [Required]
k8s.io/apimachinery/pkg/runtime.RawExtension

Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.

PluginSet

Appears in:

PluginSet specifies enabled and disabled plugins for an extension point. If an array is empty, missing, or nil, default plugins at that extension point will be used.

FieldDescription
enabled [Required]
[]Plugin

Enabled specifies plugins that should be enabled in addition to default plugins. If the default plugin is also configured in the scheduler config file, the weight of plugin will be overridden accordingly. These are called after default plugins and in the same order specified here.

disabled [Required]
[]Plugin

Disabled specifies default plugins that should be disabled. When all default plugins need to be disabled, an array containing only one "*" should be provided.

Plugins

Appears in:

Plugins include multiple extension points. When specified, the list of plugins for a particular extension point are the only ones enabled. If an extension point is omitted from the config, then the default set of plugins is used for that extension point. Enabled plugins are called in the order specified here, after default plugins. If they need to be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.

FieldDescription
queueSort [Required]
PluginSet

QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.

preFilter [Required]
PluginSet

PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.

filter [Required]
PluginSet

Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.

postFilter [Required]
PluginSet

PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.

preScore [Required]
PluginSet

PreScore is a list of plugins that are invoked before scoring.

score [Required]
PluginSet

Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.

reserve [Required]
PluginSet

Reserve is a list of plugins invoked when reserving/unreserving resources after a node is assigned to run the pod.

permit [Required]
PluginSet

Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.

preBind [Required]
PluginSet

PreBind is a list of plugins that should be invoked before a pod is bound.

bind [Required]
PluginSet

Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.

postBind [Required]
PluginSet

PostBind is a list of plugins that should be invoked after a pod is successfully bound.

multiPoint [Required]
PluginSet

MultiPoint is a simplified config section to enable plugins for all valid extension points.

PodTopologySpreadConstraintsDefaulting

(Alias of string)

Appears in:

PodTopologySpreadConstraintsDefaulting defines how to set default constraints for the PodTopologySpread plugin.

RequestedToCapacityRatioParam

Appears in:

RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters

FieldDescription
shape [Required]
[]UtilizationShapePoint

Shape is a list of points defining the scoring function shape.

ResourceSpec

Appears in:

ResourceSpec represents a single resource.

FieldDescription
name [Required]
string

Name of the resource.

weight [Required]
int64

Weight of the resource.

ScoringStrategy

Appears in:

ScoringStrategy define ScoringStrategyType for node resource plugin

FieldDescription
type [Required]
ScoringStrategyType

Type selects which strategy to run.

resources [Required]
[]ResourceSpec

Resources to consider when scoring. The default resource set includes "cpu" and "memory" with an equal weight. Allowed weights go from 1 to 100. Weight defaults to 1 if not specified or explicitly set to 0.

requestedToCapacityRatio [Required]
RequestedToCapacityRatioParam

Arguments specific to RequestedToCapacityRatio strategy.

ScoringStrategyType

(Alias of string)

Appears in:

ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.

UtilizationShapePoint

Appears in:

UtilizationShapePoint represents single point of priority function shape.

FieldDescription
utilization [Required]
int32

Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.

score [Required]
int32

Score assigned to given utilization (y axis). Valid values are 0 to 10.

11 - kube-scheduler Configuration (v1beta3)

Resource Types

ClientConnectionConfiguration

Appears in:

ClientConnectionConfiguration contains details for constructing a client.

FieldDescription
kubeconfig [Required]
string

kubeconfig is the path to a KubeConfig file.

acceptContentTypes [Required]
string

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular client.

contentType [Required]
string

contentType is the content type used when sending data to the server from this client.

qps [Required]
float32

qps controls the number of queries per second allowed for this connection.

burst [Required]
int32

burst allows extra queries to accumulate when a client is exceeding its rate.

DebuggingConfiguration

Appears in:

DebuggingConfiguration holds configuration for Debugging related features.

FieldDescription
enableProfiling [Required]
bool

enableProfiling enables profiling via web interface host:port/debug/pprof/

enableContentionProfiling [Required]
bool

enableContentionProfiling enables lock contention profiling, if enableProfiling is true.

FormatOptions

Appears in:

FormatOptions contains options for the different logging formats.

FieldDescription
json [Required]
JSONOptions

[Experimental] JSON contains options for logging format "json".

JSONOptions

Appears in:

JSONOptions contains options for logging format "json".

FieldDescription
splitStream [Required]
bool

[Experimental] SplitStream redirects error messages to stderr while info messages go to stdout, with buffering. The default is to write both to stdout, without buffering.

infoBufferSize [Required]
k8s.io/apimachinery/pkg/api/resource.QuantityValue

[Experimental] InfoBufferSize sets the size of the info stream when using split streams. The default is zero, which disables buffering.

LeaderElectionConfiguration

Appears in:

LeaderElectionConfiguration defines the configuration of leader election clients for components that can run with leader election enabled.

FieldDescription
leaderElect [Required]
bool

leaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated components for high availability.

leaseDuration [Required]
meta/v1.Duration

leaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

renewDeadline [Required]
meta/v1.Duration

renewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

retryPeriod [Required]
meta/v1.Duration

retryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

resourceLock [Required]
string

resourceLock indicates the resource object type that will be used to lock during leader election cycles.

resourceName [Required]
string

resourceName indicates the name of resource object that will be used to lock during leader election cycles.

resourceNamespace [Required]
string

resourceName indicates the namespace of resource object that will be used to lock during leader election cycles.

LoggingConfiguration

Appears in:

LoggingConfiguration contains logging options Refer Logs Options for more information.

FieldDescription
format [Required]
string

Format Flag specifies the structure of log messages. default value of format is text

flushFrequency [Required]
time.Duration

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log flushes. Ignored if the selected logging backend writes log messages without buffering.

verbosity [Required]
uint32

Verbosity is the threshold that determines which log messages are logged. Default is zero which logs only the most important messages. Higher values enable additional messages. Error messages are always logged.

vmodule [Required]
VModuleConfiguration

VModule overrides the verbosity threshold for individual files. Only supported for "text" log format.

options [Required]
FormatOptions

[Experimental] Options holds additional parameters that are specific to the different logging formats. Only the options for the selected format get used, but all of them get validated.

VModuleConfiguration

(Alias of []k8s.io/component-base/config/v1alpha1.VModuleItem)

Appears in:

VModuleConfiguration is a collection of individual file names or patterns and the corresponding verbosity threshold.

DefaultPreemptionArgs

DefaultPreemptionArgs holds arguments used to configure the DefaultPreemption plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
DefaultPreemptionArgs
minCandidateNodesPercentage [Required]
int32

MinCandidateNodesPercentage is the minimum number of candidates to shortlist when dry running preemption as a percentage of number of nodes. Must be in the range [0, 100]. Defaults to 10% of the cluster size if unspecified.

minCandidateNodesAbsolute [Required]
int32

MinCandidateNodesAbsolute is the absolute minimum number of candidates to shortlist. The likely number of candidates enumerated for dry running preemption is given by the formula: numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute) We say "likely" because there are other factors such as PDB violations that play a role in the number of candidates shortlisted. Must be at least 0 nodes. Defaults to 100 nodes if unspecified.

InterPodAffinityArgs

InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
InterPodAffinityArgs
hardPodAffinityWeight [Required]
int32

HardPodAffinityWeight is the scoring weight for existing pods with a matching hard affinity to the incoming pod.

KubeSchedulerConfiguration

KubeSchedulerConfiguration configures a scheduler

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
KubeSchedulerConfiguration
parallelism [Required]
int32

Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16

leaderElection [Required]
LeaderElectionConfiguration

LeaderElection defines the configuration of leader election client.

clientConnection [Required]
ClientConnectionConfiguration

ClientConnection specifies the kubeconfig file and client connection settings for the proxy server to use when communicating with the apiserver.

DebuggingConfiguration [Required]
DebuggingConfiguration
(Members of DebuggingConfiguration are embedded into this type.)

DebuggingConfiguration holds configuration for Debugging related features TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration

percentageOfNodesToScore [Required]
int32

PercentageOfNodesToScore is the percentage of all nodes that once found feasible for running a pod, the scheduler stops its search for more feasible nodes in the cluster. This helps improve scheduler's performance. Scheduler always tries to find at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. Example: if the cluster size is 500 nodes and the value of this flag is 30, then scheduler stops finding further feasible nodes once it finds 150 feasible ones. When the value is 0, default percentage (5%--50% based on the size of the cluster) of the nodes will be scored.

podInitialBackoffSeconds [Required]
int64

PodInitialBackoffSeconds is the initial backoff for unschedulable pods. If specified, it must be greater than 0. If this value is null, the default value (1s) will be used.

podMaxBackoffSeconds [Required]
int64

PodMaxBackoffSeconds is the max backoff for unschedulable pods. If specified, it must be greater than podInitialBackoffSeconds. If this value is null, the default value (10s) will be used.

profiles [Required]
[]KubeSchedulerProfile

Profiles are scheduling profiles that kube-scheduler supports. Pods can choose to be scheduled under a particular profile by setting its associated scheduler name. Pods that don't specify any scheduler name are scheduled with the "default-scheduler" profile, if present here.

extenders [Required]
[]Extender

Extenders are the list of scheduler extenders, each holding the values of how to communicate with the extender. These extenders are shared by all scheduler profiles.

NodeAffinityArgs

NodeAffinityArgs holds arguments to configure the NodeAffinity plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
NodeAffinityArgs
addedAffinity
core/v1.NodeAffinity

AddedAffinity is applied to all Pods additionally to the NodeAffinity specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes match). When AddedAffinity is used, some Pods with affinity requirements that match a specific Node (such as Daemonset Pods) might remain unschedulable.

NodeResourcesBalancedAllocationArgs

NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
NodeResourcesBalancedAllocationArgs
resources [Required]
[]ResourceSpec

Resources to be managed, the default is "cpu" and "memory" if not specified.

NodeResourcesFitArgs

NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
NodeResourcesFitArgs
ignoredResources [Required]
[]string

IgnoredResources is the list of resources that NodeResources fit filter should ignore. This doesn't apply to scoring.

ignoredResourceGroups [Required]
[]string

IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. e.g. if group is ["example.com"], it will ignore all resource names that begin with "example.com", such as "example.com/aaa" and "example.com/bbb". A resource group name can't contain '/'. This doesn't apply to scoring.

scoringStrategy [Required]
ScoringStrategy

ScoringStrategy selects the node resource scoring strategy. The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.

PodTopologySpreadArgs

PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
PodTopologySpreadArgs
defaultConstraints
[]core/v1.TopologySpreadConstraint

DefaultConstraints defines topology spread constraints to be applied to Pods that don't define any in pod.spec.topologySpreadConstraints. .defaultConstraints[*].labelSelectors must be empty, as they are deduced from the Pod's membership to Services, ReplicationControllers, ReplicaSets or StatefulSets. When not empty, .defaultingType must be "List".

defaultingType
PodTopologySpreadConstraintsDefaulting

DefaultingType determines how .defaultConstraints are deduced. Can be one of "System" or "List".

  • "System": Use kubernetes defined constraints that spread Pods among Nodes and Zones.
  • "List": Use constraints defined in .defaultConstraints.

Defaults to "System".

VolumeBindingArgs

VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.

FieldDescription
apiVersion
string
kubescheduler.config.k8s.io/v1beta3
kind
string
VolumeBindingArgs
bindTimeoutSeconds [Required]
int64

BindTimeoutSeconds is the timeout in seconds in volume binding operation. Value must be non-negative integer. The value zero indicates no waiting. If this value is nil, the default value (600) will be used.

shape
[]UtilizationShapePoint

Shape specifies the points defining the score function shape, which is used to score nodes based on the utilization of statically provisioned PVs. The utilization is calculated by dividing the total requested storage of the pod by the total capacity of feasible PVs on each node. Each point contains utilization (ranges from 0 to 100) and its associated score (ranges from 0 to 10). You can turn the priority by specifying different scores for different utilization numbers. The default shape points are:

  1. 0 for 0 utilization
  2. 10 for 100 utilization All points must be sorted in increasing order by utilization.

Extender

Appears in:

Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, it is assumed that the extender chose not to provide that extension.

FieldDescription
urlPrefix [Required]
string

URLPrefix at which the extender is available

filterVerb [Required]
string

Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.

preemptVerb [Required]
string

Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.

prioritizeVerb [Required]
string

Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.

weight [Required]
int64

The numeric multiplier for the node scores that the prioritize call generates. The weight should be a positive integer

bindVerb [Required]
string

Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender can implement this function.

enableHTTPS [Required]
bool

EnableHTTPS specifies whether https should be used to communicate with the extender

tlsConfig [Required]
ExtenderTLSConfig

TLSConfig specifies the transport layer security config

httpTimeout [Required]
meta/v1.Duration

HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize timeout is ignored, k8s/other extenders priorities are used to select the node.

nodeCacheCapable [Required]
bool

NodeCacheCapable specifies that the extender is capable of caching node information, so the scheduler should only send minimal information about the eligible nodes assuming that the extender already cached full details of all nodes in the cluster

managedResources
[]ExtenderManagedResource

ManagedResources is a list of extended resources that are managed by this extender.

  • A pod will be sent to the extender on the Filter, Prioritize and Bind (if the extender is the binder) phases iff the pod requests at least one of the extended resources in this list. If empty or unspecified, all pods will be sent to this extender.
  • If IgnoredByScheduler is set to true for a resource, kube-scheduler will skip checking the resource in predicates.
ignorable [Required]
bool

Ignorable specifies if the extender is ignorable, i.e. scheduling should not fail when the extender returns an error or is not reachable.

ExtenderManagedResource

Appears in:

ExtenderManagedResource describes the arguments of extended resources managed by an extender.

FieldDescription
name [Required]
string

Name is the extended resource name.

ignoredByScheduler [Required]
bool

IgnoredByScheduler indicates whether kube-scheduler should ignore this resource when applying predicates.

ExtenderTLSConfig

Appears in:

ExtenderTLSConfig contains settings to enable TLS with extender

FieldDescription
insecure [Required]
bool

Server should be accessed without verifying the TLS certificate. For testing only.

serverName [Required]
string

ServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname used to contact the server is used.

certFile [Required]
string

Server requires TLS client certificate authentication

keyFile [Required]
string

Server requires TLS client certificate authentication

caFile [Required]
string

Trusted root certificates for server

certData [Required]
[]byte

CertData holds PEM-encoded bytes (typically read from a client certificate file). CertData takes precedence over CertFile

keyData [Required]
[]byte

KeyData holds PEM-encoded bytes (typically read from a client certificate key file). KeyData takes precedence over KeyFile

caData [Required]
[]byte

CAData holds PEM-encoded bytes (typically read from a root certificates bundle). CAData takes precedence over CAFile

KubeSchedulerProfile

Appears in:

KubeSchedulerProfile is a scheduling profile.

FieldDescription
schedulerName [Required]
string

SchedulerName is the name of the scheduler associated to this profile. If SchedulerName matches with the pod's "spec.schedulerName", then the pod is scheduled with this profile.

plugins [Required]
Plugins

Plugins specify the set of plugins that should be enabled or disabled. Enabled plugins are the ones that should be enabled in addition to the default plugins. Disabled plugins are any of the default plugins that should be disabled. When no enabled or disabled plugin is specified for an extension point, default plugins for that extension point will be used if there is any. If a QueueSort plugin is specified, the same QueueSort Plugin and PluginConfig must be specified for all profiles.

pluginConfig [Required]
[]PluginConfig

PluginConfig is an optional set of custom plugin arguments for each plugin. Omitting config args for a plugin is equivalent to using the default config for that plugin.

Plugin

Appears in:

Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.

FieldDescription
name [Required]
string

Name defines the name of plugin

weight [Required]
int32

Weight defines the weight of plugin, only used for Score plugins.

PluginConfig

Appears in:

PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. It is up to the plugin to process these Args.

FieldDescription
name [Required]
string

Name defines the name of plugin being configured

args [Required]
k8s.io/apimachinery/pkg/runtime.RawExtension

Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.

PluginSet

Appears in:

PluginSet specifies enabled and disabled plugins for an extension point. If an array is empty, missing, or nil, default plugins at that extension point will be used.

FieldDescription
enabled [Required]
[]Plugin

Enabled specifies plugins that should be enabled in addition to default plugins. If the default plugin is also configured in the scheduler config file, the weight of plugin will be overridden accordingly. These are called after default plugins and in the same order specified here.

disabled [Required]
[]Plugin

Disabled specifies default plugins that should be disabled. When all default plugins need to be disabled, an array containing only one "*" should be provided.

Plugins

Appears in:

Plugins include multiple extension points. When specified, the list of plugins for a particular extension point are the only ones enabled. If an extension point is omitted from the config, then the default set of plugins is used for that extension point. Enabled plugins are called in the order specified here, after default plugins. If they need to be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.

FieldDescription
queueSort [Required]
PluginSet

QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.

preFilter [Required]
PluginSet

PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.

filter [Required]
PluginSet

Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.

postFilter [Required]
PluginSet

PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.

preScore [Required]
PluginSet

PreScore is a list of plugins that are invoked before scoring.

score [Required]
PluginSet

Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.

reserve [Required]
PluginSet

Reserve is a list of plugins invoked when reserving/unreserving resources after a node is assigned to run the pod.

permit [Required]
PluginSet

Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.

preBind [Required]
PluginSet

PreBind is a list of plugins that should be invoked before a pod is bound.

bind [Required]
PluginSet

Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.

postBind [Required]
PluginSet

PostBind is a list of plugins that should be invoked after a pod is successfully bound.

multiPoint [Required]
PluginSet

MultiPoint is a simplified config section to enable plugins for all valid extension points. Plugins enabled through MultiPoint will automatically register for every individual extension point the plugin has implemented. Disabling a plugin through MultiPoint disables that behavior. The same is true for disabling "*" through MultiPoint (no default plugins will be automatically registered). Plugins can still be disabled through their individual extension points.

In terms of precedence, plugin config follows this basic hierarchy

  1. Specific extension points
  2. Explicitly configured MultiPoint plugins
  3. The set of default plugins, as MultiPoint plugins This implies that a higher precedence plugin will run first and overwrite any settings within MultiPoint. Explicitly user-configured plugins also take a higher precedence over default plugins. Within this hierarchy, an Enabled setting takes precedence over Disabled. For example, if a plugin is set in both multiPoint.Enabled and multiPoint.Disabled, the plugin will be enabled. Similarly, including multiPoint.Disabled = '*' and multiPoint.Enabled = pluginA will still register that specific plugin through MultiPoint. This follows the same behavior as all other extension point configurations.

PodTopologySpreadConstraintsDefaulting

(Alias of string)

Appears in:

PodTopologySpreadConstraintsDefaulting defines how to set default constraints for the PodTopologySpread plugin.

RequestedToCapacityRatioParam

Appears in:

RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters

FieldDescription
shape [Required]
[]UtilizationShapePoint

Shape is a list of points defining the scoring function shape.

ResourceSpec

Appears in:

ResourceSpec represents a single resource.

FieldDescription
name [Required]
string

Name of the resource.

weight [Required]
int64

Weight of the resource.

ScoringStrategy

Appears in:

ScoringStrategy define ScoringStrategyType for node resource plugin

FieldDescription
type [Required]
ScoringStrategyType

Type selects which strategy to run.

resources [Required]
[]ResourceSpec

Resources to consider when scoring. The default resource set includes "cpu" and "memory" with an equal weight. Allowed weights go from 1 to 100. Weight defaults to 1 if not specified or explicitly set to 0.

requestedToCapacityRatio [Required]
RequestedToCapacityRatioParam

Arguments specific to RequestedToCapacityRatio strategy.

ScoringStrategyType

(Alias of string)

Appears in:

ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.

UtilizationShapePoint

Appears in:

UtilizationShapePoint represents single point of priority function shape.

FieldDescription
utilization [Required]
int32

Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.

score [Required]
int32

Score assigned to given utilization (y axis). Valid values are 0 to 10.

12 - kubeadm Configuration (v1beta2)

Overview

Package v1beta2 defines the v1beta2 version of the kubeadm configuration file format. This version improves on the v1beta1 format by fixing some minor issues and adding a few new fields.

A list of changes since v1beta1:

  • "certificateKey" field is added to InitConfiguration and JoinConfiguration.
  • "ignorePreflightErrors" field is added to the NodeRegistrationOptions.
  • The JSON "omitempty" tag is used in a more places where appropriate.
  • The JSON "omitempty" tag of the "taints" field (inside NodeRegistrationOptions) is removed.

See the Kubernetes 1.15 changelog for further details.

Migration from old kubeadm config versions

Please convert your v1beta1 configuration files to v1beta2 using the "kubeadm config migrate" command of kubeadm v1.15.x (conversion from older releases of kubeadm config files requires older release of kubeadm as well e.g.

  • kubeadm v1.11 should be used to migrate v1alpha1 to v1alpha2; kubeadm v1.12 should be used to translate v1alpha2 to v1alpha3;
  • kubeadm v1.13 or v1.14 should be used to translate v1alpha3 to v1beta1)

Nevertheless, kubeadm v1.15.x will support reading from v1beta1 version of the kubeadm config file format.

Basics

The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the configuration options defined in the kubeadm config file are also available as command line flags, but only the most common/simple use case are supported with this approach.

A kubeadm config file could contain multiple configuration types separated using three dashes (---).

kubeadm supports the following configuration types:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration

To print the defaults for "init" and "join" actions use the following commands:

kubeadm config print init-defaults
kubeadm config print join-defaults

The list of configuration types that must be included in a configuration file depends by the action you are performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).

If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults provided by kubeadm includes also enforcing consistency of values across components when required (e.g. --cluster-cidr flag on controller manager and clusterCIDR on kube-proxy).

Users are always allowed to override default values, with the only exception of a small subset of setting with relevance for security (e.g. enforce authorization-mode Node and RBAC on API server)

If the user provides a configuration types that is not expected for the action you are performing, kubeadm will ignore those types and print a warning.

Kubeadm init configuration types

When executing kubeadm init with the --config option, the following configuration types could be used: InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one between InitConfiguration and ClusterConfiguration is mandatory.

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
  ...
nodeRegistration:
  ...

The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm is executed, including:

  • nodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node ip).

  • apiServer, that represents the endpoint of the instance of the API server to be deployed on this node; use it e.g. to customize the API server advertise address.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
    ...
etcd:
    ...
apiServer:
  extraArgs:
    ...
  extraVolumes:
    ...
...

The ClusterConfiguration type should be used to configure cluster-wide settings, including settings for:

  • Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize pod subnet or services subnet.

  • Etcd configurations; use it e.g. to customize the local etcd or to configure the API server for using an external etcd cluster.

  • kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane components by adding customized setting or overriding kubeadm default settings.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
  ...

The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://pkg.go.dev/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration for kube proxy official documentation.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
  ...

The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration for kubelet official documentation.

Here is a fully populated example of a single YAML file containing multiple configuration types to be used during a kubeadm init run.

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
  - token: "9a08jv.c0izixklcxtmnze7"
    description: "kubeadm bootstrap token"
    ttl: "24h"
  - token: "783bde.3f89s0fje9f38fhf"
    description: "another bootstrap token"
    usages:
      - authentication
      - signing
    groups:
      - system:bootstrappers:kubeadm:default-node-token
nodeRegistration:
  name: "ec2-10-100-0-1"
  criSocket: "/var/run/dockershim.sock"
  taints:
    - key: "kubeadmNode"
      value: "someValue"
      effect: "NoSchedule"
  kubeletExtraArgs:
    v: 4
  ignorePreflightErrors:
    - IsPrivilegedUser
localAPIEndpoint:
  advertiseAddress: "10.100.0.1"
  bindPort: 6443
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  # one of local or external
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    dataDir: "/var/lib/etcd"
    extraArgs:
      listen-client-urls: "http://10.100.0.1:2379"
    serverCertSANs:
      -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    peerCertSANs:
      - "10.100.0.1"
  # external:
  #   endpoints:
  #     - "10.100.0.1:2379"
  #     - "10.100.0.2:2379"
  #   caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
  #   certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
  #   keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/24"
  dnsDomain: "cluster.local"
kubernetesVersion: "v1.12.0"
controlPlaneEndpoint: "10.100.0.1:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
  certSANs:
    - "10.100.1.1"
    - "ec2-10-100-0-1.compute-1.amazonaws.com"
  timeoutForControlPlane: 4m0s
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
scheduler:
  extraArgs:
    address: "10.100.0.1"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
useHyperKubeImage: false
clusterName: "example-cluster"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# kubelet specific options here
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
# kube-proxy specific options here

Kubeadm join configuration types

When executing kubeadm join with the --config option, the JoinConfiguration type should be provided.

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
  ...

The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join are the discovery method used for accessing the cluster info and all the setting which are specific to the node where kubeadm is executed, including:

  • NodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node IP).

  • APIEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node.

Resource Types

ClusterConfiguration

ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta2
kind
string
ClusterConfiguration
etcd [Required]
Etcd

etcd holds configuration for etcd.

networking [Required]
Networking

networking holds configuration for the networking topology of the cluster.

kubernetesVersion [Required]
string

kubernetesVersion is the target version of the control plane.

controlPlaneEndpoint [Required]
string

controlPlaneEndpoint sets a stable IP address or DNS name for the control plane; it can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port. In case the controlPlaneEndpoint is not specified, the advertiseAddress + bindPort are used; in case the controlPlaneEndpoint is specified but without a TCP port, the bindPort is used. Possible usages are:

  • In a cluster with more than one control plane instances, this field should be assigned the address of the external load balancer in front of the control plane instances.
  • In environments with enforced node recycling, the controlPlaneEndpoint could be used for assigning a stable DNS to the control plane.
apiServer [Required]
APIServer

apiServer contains extra settings for the API server.

controllerManager [Required]
ControlPlaneComponent

controllerManager contains extra settings for the controller manager.

scheduler [Required]
ControlPlaneComponent

scheduler contains extra settings for the scheduler.

dns [Required]
DNS

dns defines the options for the DNS add-on installed in the cluster.

certificatesDir [Required]
string

certificatesDir specifies where to store or look for all required certificates.

imageRepository [Required]
string

imageRepository sets the container registry to pull images from. If empty, k8s.gcr.io will be used by default; in case of kubernetes version is a CI build (kubernetes version starts with ci/) gcr.io/k8s-staging-ci-images is used as a default for control plane components and for kube-proxy, while k8s.gcr.io will be used for all the other images.

useHyperKubeImage [Required]
bool

useHyperKubeImage controls if hyperkube should be used for Kubernetes components instead of their respective separate images. DEPRECATED: As hyperkube is itself deprecated, this fields is too. It will be removed in future kubeadm config versions, kubeadm will print multiple warnings when this set to true, and at some point it may become ignored.

featureGates [Required]
map[string]bool

featureGates contains the feature gates enabled by the user.

clusterName [Required]
string

The cluster name.

ClusterStatus

ClusterStatus contains the cluster status. The ClusterStatus will be stored in the kubeadm-config ConfigMap in the cluster, and then updated by kubeadm when additional control plane instance joins or leaves the cluster.

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta2
kind
string
ClusterStatus
apiEndpoints [Required]
map[string]github.com/tengqm/kubeconfig/config/kubeadm/v1beta2.APIEndpoint

apiEndpoints currently available in the cluster, one for each control plane/API server instance. The key of the map is the IP of the host's default interface.

InitConfiguration

InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime information.

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta2
kind
string
InitConfiguration
bootstrapTokens [Required]
[]BootstrapToken

bootstrapTokens is respected at kubeadm init time and describes a set of bootstrap tokens to create. This information IS NOT uploaded to the kubeadm cluster ConfigMap, partly because of its sensitive nature.

nodeRegistration [Required]
NodeRegistrationOptions

nodeRegistration holds fields that relate to registering the new control-plane node to the cluster.

localAPIEndpoint [Required]
APIEndpoint

localAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node. In HA setups, this differs from ClusterConfiguration.controlPlaneEndpoint in the sense that ControlPlaneEndpoint is the global endpoint for the cluster, which then load-balances the requests to each individual API server. This configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process fails you may set the desired value here.

certificateKey [Required]
string

certificateKey sets the key with which certificates and keys are encrypted prior to being uploaded in a secret in the cluster during the uploadcerts init phase.

JoinConfiguration

JoinConfiguration contains elements describing a particular node.

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta2
kind
string
JoinConfiguration
nodeRegistration [Required]
NodeRegistrationOptions

nodeRegistration holds fields that relate to registering the new control-plane node to the cluster

caCertPath [Required]
string

caCertPath is the path to the SSL certificate authority used to secure comunications between a node and the control-plane. Defaults to "/etc/kubernetes/pki/ca.crt".

discovery [Required]
Discovery

discovery specifies the options for the kubelet to use during the TLS bootstrap process.

controlPlane [Required]
JoinControlPlane

controlPlane defines the additional control plane instance to be deployed on the joining node. If nil, no additional control plane instance will be deployed.

APIEndpoint

Appears in:

APIEndpoint struct contains elements of API server instance deployed on a node.

FieldDescription
advertiseAddress [Required]
string

advertiseAddress sets the IP address for the API server to advertise.

bindPort [Required]
int32

bindPort sets the secure port for the API Server to bind to. Defaults to 6443.

APIServer

Appears in:

APIServer holds settings necessary for API server deployments in the cluster.

FieldDescription
ControlPlaneComponent [Required]
ControlPlaneComponent
(Members of ControlPlaneComponent are embedded into this type.) No description provided.
certSANs [Required]
[]string

certSANs sets extra Subject Alternative Names (SANs) for the API Server signing certificate.

timeoutForControlPlane [Required]
meta/v1.Duration

timeoutForControlPlane controls the timeout that we wait for the API server to appear.

BootstrapToken

Appears in:

BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

FieldDescription
token [Required]
BootstrapTokenString

token is used for establishing bidirectional trust between nodes and control-planes. Used for joining nodes in the cluster.

description [Required]
string

description sets a human-friendly message why this token exists and what it's used for, so other administrators can know its purpose.

ttl [Required]
meta/v1.Duration

ttl defines the time to live for this token. Defaults to '24h'. expires and ttl are mutually exclusive.

expires [Required]
meta/v1.Time

expires specifies the timestamp when this token expires. Defaults to being set dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

usages [Required]
[]string

usages describes the ways in which this token can be used. Can by default be used for establishing bidirectional trust, but that can be changed here.

groups [Required]
[]string

groups specifies the extra groups that this token will authenticate as when/if used for authentication.

BootstrapTokenDiscovery

Appears in:

BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery

FieldDescription
token [Required]
string

token is a token used to validate cluster information fetched from the control-plane.

apiServerEndpoint [Required]
string

apiServerEndpoint is an IP or domain name to the API server from which information will be fetched.

caCertHashes [Required]
[]string

caCertHashes specifies a set of public key pins to verify when token-based discovery is used. The root CA found during discovery must match one of these values. Specifying an empty set disables root CA pinning, which can be unsafe. Each hash is specified as ":", where the only currently supported type is "sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI) object in DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.

unsafeSkipCAVerification [Required]
bool

unsafeSkipCAVerification allows token-based discovery without CA verification via caCertHashes. This can weaken the security of kubeadm since other nodes can impersonate the control-plane.

BootstrapTokenString

Appears in:

BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used for both validation of the practically of the API server from a joining node's point of view and as an authentication method for the node in the bootstrap phase of "kubeadm join". This token is and should be short-lived

FieldDescription
- [Required]
string
No description provided.
- [Required]
string
No description provided.

ControlPlaneComponent

Appears in:

ControlPlaneComponent holds settings common to control plane component of the cluster

FieldDescription
extraArgs [Required]
map[string]string

extraArgs is an extra set of flags to pass to a control plane component. A key in this map is the flag name as it appears on the command line except without leading dash(es).

extraVolumes [Required]
[]HostPathMount

extraVolumes is an extra set of host volumes mounted to the control plane component.

DNS

Appears in:

DNS defines the DNS addon that should be used in the cluster

FieldDescription
type [Required]
DNSAddOnType

type defines the DNS add-on to be used.

ImageMeta [Required]
ImageMeta
(Members of ImageMeta are embedded into this type.)

ImageMeta allows to customize the image used for the DNS component

DNSAddOnType

(Alias of string)

Appears in:

DNSAddOnType defines string identifying DNS add-on types.

Discovery

Appears in:

Discovery specifies the options for the kubelet to use during the TLS Bootstrap process

FieldDescription
bootstrapToken [Required]
BootstrapTokenDiscovery

bootstrapToken is used to set the options for bootstrap token based discovery. bootstrapToken and file are mutually exclusive.

file [Required]
FileDiscovery

file is used to specify a file or URL to a kubeconfig file from which to load cluster information. bootstrapToken and file are mutually exclusive.

tlsBootstrapToken [Required]
string

tlsBootstrapToken is a token used for TLS bootstrapping. If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, but can be overridden. If file is set, this field must be set in case the KubeConfigFile does not contain any other authentication information.

timeout [Required]
meta/v1.Duration

timeout modifies the discovery timeout.

Etcd

Appears in:

Etcd contains elements describing Etcd configuration.

FieldDescription
local [Required]
LocalEtcd

local provides configuration knobs for configuring the local etcd instance. local and external are mutually exclusive.

external [Required]
ExternalEtcd

external describes how to connect to an external etcd cluster. local and external are mutually exclusive.

ExternalEtcd

Appears in:

ExternalEtcd describes an external etcd cluster. Kubeadm has no knowledge of where certificate files live and they must be supplied.

FieldDescription
endpoints [Required]
[]string

endpoints of etcd members.

caFile [Required]
string

caFile is an SSL Certificate Authority (CA) file used to secure etcd communication. Required if using a TLS connection.

certFile [Required]
string

certFile is an SSL certification file used to secure etcd communication. Required if using a TLS connection.

keyFile [Required]
string

keyFile is an SSL key file used to secure etcd communication. Required if using a TLS connection.

FileDiscovery

Appears in:

FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information

FieldDescription
kubeConfigPath [Required]
string

kubeConfigPath is used to specify the actual file path or URL to the kubeconfig file from which to load cluster information.

HostPathMount

Appears in:

HostPathMount contains elements describing volumes that are mounted from the host.

FieldDescription
name [Required]
string

name of the volume inside the Pod template.

hostPath [Required]
string

hostPath is the path in the host that will be mounted inside the Pod.

mountPath [Required]
string

mountPathis the path inside the Pod where hostPath volume will be mounted.

readOnly [Required]
bool

readOnly controls write access to the volume.

pathType [Required]
core/v1.HostPathType

pathType is the type of the HostPath.

ImageMeta

Appears in:

ImageMeta allows to customize the image used for components that are not originated from the Kubernetes/Kubernetes release process

FieldDescription
imageRepository [Required]
string

imageRepository sets the container registry to pull images from. If not set, the imageRepository defined in ClusterConfiguration will be used.

imageTag [Required]
string

imageTag allows for specifying a tag for the image. In case this value is set, kubeadm does not change automatically the version of the above components during upgrades.

JoinControlPlane

Appears in:

JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node.

FieldDescription
localAPIEndpoint [Required]
APIEndpoint

localAPIEndpoint represents the endpoint of the API server instance to be deployed on this node.

certificateKey [Required]
string

certificateKey is the key that is used for decryption of certificates after they are downloaded from the secret upon joining a new control plane node. The corresponding encryption key is in the InitConfiguration.

LocalEtcd

Appears in:

LocalEtcd describes that kubeadm should run an etcd cluster locally.

FieldDescription
ImageMeta [Required]
ImageMeta
(Members of ImageMeta are embedded into this type.)

ImageMeta allows to customize the container used for etcd.

dataDir [Required]
string

dataDir is the directory etcd will place its data. Defaults to "/var/lib/etcd".

extraArgs [Required]
map[string]string

extraArgs are extra arguments provided to the etcd binary when run inside a static pod. A key in this map is the flag name as it appears on the command line except without leading dash(es).

serverCertSANs [Required]
[]string

serverCertSANs sets extra Subject Alternative Names (SANs) for the etcd server signing certificate.

peerCertSANs [Required]
[]string

peerCertSANs sets extra Subject Alternative Names (SANs) for the etcd peer signing certificate.

Networking

Appears in:

Networking contains elements describing cluster's networking configuration

FieldDescription
serviceSubnet [Required]
string

serviceSubnet is the subnet used by kubernetes Services. Defaults to "10.96.0.0/12".

podSubnet [Required]
string

podSubnet is the subnet used by Pods.

dnsDomain [Required]
string

dnsDomain is the DNS domain used by kubernetes Services. Defaults to "cluster.local".

NodeRegistrationOptions

Appears in:

NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via "kubeadm init" or "kubeadm join".

FieldDescription
name [Required]
string

name is the .Metadata.Name field of the Node API object that will be created in this kubeadm init or kubeadm join operation. This field is also used in the CommonName field of the kubelet's client certificate to the API server. Defaults to the hostname of the node if not provided.

criSocket [Required]
string

criSocket is used to retrieve container runtime information. This information will be annotated to the Node API object, for later re-use.

taints [Required]
[]core/v1.Taint

taints specifies the taints the Node API object should be registered with. If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted with a control-plane taint for control-plane nodes. If you don't want to taint your control-plane node, set this field to an empty list, i.e. taints: [], in the YAML file. This field is solely used for Node registration.

kubeletExtraArgs [Required]
map[string]string

kubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the 'kubelet-config-1.X' ConfigMap. Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. A key in this map is the flag name as it appears on the command line except without leading dash(es).

ignorePreflightErrors [Required]
[]string

ignorePreflightErrors provides a list of pre-flight errors to be ignored when the current node is registered.

13 - kubeadm Configuration (v1beta3)

Overview

Package v1beta3 defines the v1beta3 version of the kubeadm configuration file format. This version improves on the v1beta2 format by fixing some minor issues and adding a few new fields.

A list of changes since v1beta2:

  • The deprecated "ClusterConfiguration.useHyperKubeImage" field has been removed. Kubeadm no longer supports the hyperkube image.
  • The "ClusterConfiguration.DNS.Type" field has been removed since CoreDNS is the only supported DNS server type by kubeadm.
  • Include "datapolicy" tags on the fields that hold secrets. This would result in the field values to be omitted when API structures are printed with klog.
  • Add "InitConfiguration.SkipPhases", "JoinConfiguration.SkipPhases" to allow skipping a list of phases during kubeadm init/join command execution.
  • Add "InitConfiguration.NodeRegistration.ImagePullPolicy" and "JoinConfiguration.NodeRegistration.ImagePullPolicy" to allow specifying the images pull policy during kubeadm "init" and "join". The value must be one of "Always", "Never" or "IfNotPresent". "IfNotPresent" is the default, which has been the existing behavior prior to this addition.
  • Add "InitConfiguration.Patches.Directory", "JoinConfiguration.Patches.Directory" to allow the user to configure a directory from which to take patches for components deployed by kubeadm.
  • Move the BootstrapToken* API and related utilities out of the "kubeadm" API group to a new group "bootstraptoken". The kubeadm API version v1beta3 no longer contains the BootstrapToken* structures.

Migration from old kubeadm config versions

  • kubeadm v1.15.x and newer can be used to migrate from v1beta1 to v1beta2.
  • kubeadm v1.22.x and newer no longer support v1beta1 and older APIs, but can be used to migrate v1beta2 to v1beta3.

Basics

The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the configuration options defined in the kubeadm config file are also available as command line flags, but only the most common/simple use case are supported with this approach.

A kubeadm config file could contain multiple configuration types separated using three dashes (---).

kubeadm supports the following configuration types:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration

To print the defaults for "init" and "join" actions use the following commands:

kubeadm config print init-defaults
kubeadm config print join-defaults

The list of configuration types that must be included in a configuration file depends by the action you are performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).

If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults provided by kubeadm includes also enforcing consistency of values across components when required (e.g. --cluster-cidr flag on controller manager and clusterCIDR on kube-proxy).

Users are always allowed to override default values, with the only exception of a small subset of setting with relevance for security (e.g. enforce authorization-mode Node and RBAC on api server)

If the user provides a configuration types that is not expected for the action you are performing, kubeadm will ignore those types and print a warning.

Kubeadm init configuration types

When executing kubeadm init with the --config option, the following configuration types could be used: InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one between InitConfiguration and ClusterConfiguration is mandatory.

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
  ...
nodeRegistration:
  ...

The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm is executed, including:

  • NodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node ip).

  • LocalAPIEndpoint, that represents the endpoint of the instance of the API server to be deployed on this node; use it e.g. to customize the API server advertise address.

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  ...
etcd:
  ...
apiServer:
  extraArgs:
    ...
  extraVolumes:
    ...
...

The ClusterConfiguration type should be used to configure cluster-wide settings, including settings for:

  • networking that holds configuration for the networking topology of the cluster; use it e.g. to customize Pod subnet or services subnet.

  • etcd: use it e.g. to customize the local etcd or to configure the API server for using an external etcd cluster.

  • kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane components by adding customized setting or overriding kubeadm default settings.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
  ...

The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://pkg.go.dev/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration for kube-proxy official documentation.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
  ...

The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration for kubelet official documentation.

Here is a fully populated example of a single YAML file containing multiple configuration types to be used during a kubeadm init run.

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- token: "9a08jv.c0izixklcxtmnze7"
  description: "kubeadm bootstrap token"
  ttl: "24h"
- token: "783bde.3f89s0fje9f38fhf"
  description: "another bootstrap token"
  usages:
  - authentication
  - signing
  groups:
  - system:bootstrappers:kubeadm:default-node-token
nodeRegistration:
  name: "ec2-10-100-0-1"
  criSocket: "/var/run/dockershim.sock"
  taints:
  - key: "kubeadmNode"
    value: "someValue"
    effect: "NoSchedule"
  kubeletExtraArgs:
    v: 4
  ignorePreflightErrors:
    - IsPrivilegedUser
  imagePullPolicy: "IfNotPresent"
localAPIEndpoint:
  advertiseAddress: "10.100.0.1"
  bindPort: 6443
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
skipPhases:
  - addon/kube-proxy
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
etcd:
  # one of local or external
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    dataDir: "/var/lib/etcd"
    extraArgs:
      listen-client-urls: "http://10.100.0.1:2379"
    serverCertSANs:
    -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    peerCertSANs:
    - "10.100.0.1"
  # external:
    # endpoints:
    # - "10.100.0.1:2379"
    # - "10.100.0.2:2379"
    # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
    # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
    # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/24"
  dnsDomain: "cluster.local"
kubernetesVersion: "v1.21.0"
controlPlaneEndpoint: "10.100.0.1:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
  certSANs:
  - "10.100.1.1"
  - "ec2-10-100-0-1.compute-1.amazonaws.com"
  timeoutForControlPlane: 4m0s
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
scheduler:
  extraArgs:
    address: "10.100.0.1"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
clusterName: "example-cluster"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# kubelet specific options here
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
# kube-proxy specific options here

Kubeadm join configuration types

When executing kubeadm join with the --config option, the JoinConfiguration type should be provided.

apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration
  ...

The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join are the discovery method used for accessing the cluster info and all the setting which are specific to the node where kubeadm is executed, including:

  • nodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node ip).

  • apiEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node.

Resource Types

ClusterConfiguration

ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta3
kind
string
ClusterConfiguration
etcd
Etcd

etcd holds the configuration for etcd.

networking
Networking

networking holds configuration for the networking topology of the cluster.

kubernetesVersion
string

kubernetesVersion is the target version of the control plane.

controlPlaneEndpoint
string

controlPlaneEndpoint sets a stable IP address or DNS name for the control plane. It can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port. In case the controlPlaneEndpoint is not specified, the advertiseAddress + bindPort are used; in case the controlPlaneEndpoint is specified but without a TCP port, the bindPort is used. Possible usages are:

  • In a cluster with more than one control plane instances, this field should be assigned the address of the external load balancer in front of the control plane instances.
  • In environments with enforced node recycling, the controlPlaneEndpoint could be used for assigning a stable DNS to the control plane.
apiServer
APIServer

apiServer contains extra settings for the API server.

controllerManager
ControlPlaneComponent

controllerManager contains extra settings for the controller manager.

scheduler
ControlPlaneComponent

scheduler contains extra settings for the scheduler.

dns
DNS

dns defines the options for the DNS add-on installed in the cluster.

certificatesDir
string

certificatesDir specifies where to store or look for all required certificates.

imageRepository
string

imageRepository sets the container registry to pull images from. If empty, k8s.gcr.io will be used by default. In case of kubernetes version is a CI build (kubernetes version starts with ci/) gcr.io/k8s-staging-ci-images will be used as a default for control plane components and for kube-proxy, while k8s.gcr.io will be used for all the other images.

featureGates
map[string]bool

featureGates contains the feature gates enabled by the user.

clusterName
string

The cluster name.

InitConfiguration

InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime information. kubeadm init-only information. These fields are solely used the first time kubeadm init runs. After that, the information in the fields IS NOT uploaded to the kubeadm-config ConfigMap that is used by kubeadm upgrade for instance. These fields must be omitempty.

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta3
kind
string
InitConfiguration
bootstrapTokens
[]BootstrapToken

bootstrapTokens is respected at kubeadm init time and describes a set of Bootstrap Tokens to create. This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature

nodeRegistration
NodeRegistrationOptions

nodeRegistration holds fields that relate to registering the new control-plane node to the cluster.

localAPIEndpoint
APIEndpoint

localAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node. In HA setups, this differs from ClusterConfiguration.controlPlaneEndpoint in the sense that controlPlaneEndpoint is the global endpoint for the cluster, which then load-balances the requests to each individual API server. This configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process fails you may set the desired value here.

certificateKey
string

certificateKey sets the key with which certificates and keys are encrypted prior to being uploaded in a Secret in the cluster during the uploadcerts init phase.

skipPhases
[]string

skipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the kubeadm init --help command. The flag "--skip-phases" takes precedence over this field.

patches
Patches

patches contains options related to applying patches to components deployed by kubeadm during kubeadm init.

JoinConfiguration

JoinConfiguration contains elements describing a particular node.

FieldDescription
apiVersion
string
kubeadm.k8s.io/v1beta3
kind
string
JoinConfiguration
nodeRegistration
NodeRegistrationOptions

nodeRegistration holds fields that relate to registering the new control-plane node to the cluster.

caCertPath
string

caCertPath is the path to the SSL certificate authority used to secure comunications between a node and the control-plane. Defaults to "/etc/kubernetes/pki/ca.crt".

discovery [Required]
Discovery

discovery specifies the options for the kubelet to use during the TLS bootstrap process.

controlPlane
JoinControlPlane

controlPlane defines the additional control plane instance to be deployed on the joining node. If nil, no additional control plane instance will be deployed.

skipPhases
[]string

skipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the kubeadm join --help command. The flag --skip-phases takes precedence over this field.

patches
Patches

patches contains options related to applying patches to components deployed by kubeadm during kubeadm join.

APIEndpoint

Appears in:

APIEndpoint struct contains elements of API server instance deployed on a node.

FieldDescription
advertiseAddress
string

advertiseAddress sets the IP address for the API server to advertise.

bindPort
int32

bindPort sets the secure port for the API Server to bind to. Defaults to 6443.

APIServer

Appears in:

APIServer holds settings necessary for API server deployments in the cluster

FieldDescription
ControlPlaneComponent [Required]
ControlPlaneComponent
(Members of ControlPlaneComponent are embedded into this type.) No description provided.
certSANs
[]string

certSANs sets extra Subject Alternative Names (SANs) for the API Server signing certificate.

timeoutForControlPlane
meta/v1.Duration

timeoutForControlPlane controls the timeout that we wait for API server to appear.

BootstrapTokenDiscovery

Appears in:

BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery

FieldDescription
token [Required]
string

token is a token used to validate cluster information fetched from the control-plane.

apiServerEndpoint
string

apiServerEndpoint is an IP or domain name to the API server from which information will be fetched.

caCertHashes
[]string

caCertHashes specifies a set of public key pins to verify when token-based discovery is used. The root CA found during discovery must match one of these values. Specifying an empty set disables root CA pinning, which can be unsafe. Each hash is specified as ":", where the only currently supported type is "sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI) object in DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.

unsafeSkipCAVerification
bool

unsafeSkipCAVerification allows token-based discovery without CA verification via caCertHashes. This can weaken the security of kubeadm since other nodes can impersonate the control-plane.

ControlPlaneComponent

Appears in:

ControlPlaneComponent holds settings common to control plane component of the cluster

FieldDescription
extraArgs
map[string]string

extraArgs is an extra set of flags to pass to the control plane component. A key in this map is the flag name as it appears on the command line except without leading dash(es).

extraVolumes
[]HostPathMount

extraVolumes is an extra set of host volumes, mounted to the control plane component.

DNS

Appears in:

DNS defines the DNS addon that should be used in the cluster

FieldDescription
ImageMeta [Required]
ImageMeta
(Members of ImageMeta are embedded into this type.)

imageMeta allows to customize the image used for the DNS component.

Discovery

Appears in:

Discovery specifies the options for the kubelet to use during the TLS Bootstrap process.

FieldDescription
bootstrapToken
BootstrapTokenDiscovery

bootstrapToken is used to set the options for bootstrap token based discovery. bootstrapToken and file are mutually exclusive.

file
FileDiscovery

file is used to specify a file or URL to a kubeconfig file from which to load cluster information. bootstrapToken and file are mutually exclusive.

tlsBootstrapToken
string

tlsBootstrapToken is a token used for TLS bootstrapping. If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, but can be overridden. If file is set, this field must be set in case the KubeConfigFile does not contain any other authentication information

timeout
meta/v1.Duration

timeout modifies the discovery timeout.

Etcd

Appears in:

Etcd contains elements describing Etcd configuration.

FieldDescription
local
LocalEtcd

local provides configuration knobs for configuring the local etcd instance. local and external are mutually exclusive.

external
ExternalEtcd

external describes how to connect to an external etcd cluster. local and external are mutually exclusive.

ExternalEtcd

Appears in:

ExternalEtcd describes an external etcd cluster. Kubeadm has no knowledge of where certificate files live and they must be supplied.

FieldDescription
endpoints [Required]
[]string

endpoints contains the list of etcd members.

caFile [Required]
string

caFile is an SSL Certificate Authority (CA) file used to secure etcd communication. Required if using a TLS connection.

certFile [Required]
string

certFile is an SSL certification file used to secure etcd communication. Required if using a TLS connection.

keyFile [Required]
string

keyFile is an SSL key file used to secure etcd communication. Required if using a TLS connection.

FileDiscovery

Appears in:

FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information.

FieldDescription
kubeConfigPath [Required]
string

kubeConfigPath is used to specify the actual file path or URL to the kubeconfig file from which to load cluster information.

HostPathMount

Appears in:

HostPathMount contains elements describing volumes that are mounted from the host.

FieldDescription
name [Required]
string

name is the name of the volume inside the Pod template.

hostPath [Required]
string

hostPath is the path in the host that will be mounted inside the Pod.

mountPath [Required]
string

mountPath is the path inside the Pod where hostPath will be mounted.

readOnly
bool

readOnly controls write access to the volume.

pathType
core/v1.HostPathType

pathType is the type of the hostPath.

ImageMeta

Appears in:

ImageMeta allows to customize the image used for components that are not originated from the Kubernetes/Kubernetes release process

FieldDescription
imageRepository
string

imageRepository sets the container registry to pull images from. If not set, the imageRepository defined in ClusterConfiguration will be used instead.

imageTag
string

imageTag allows to specify a tag for the image. In case this value is set, kubeadm does not change automatically the version of the above components during upgrades.

JoinControlPlane

Appears in:

JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node.

FieldDescription
localAPIEndpoint
APIEndpoint

localAPIEndpoint represents the endpoint of the API server instance to be deployed on this node.

certificateKey
string

certificateKey is the key that is used for decryption of certificates after they are downloaded from the secret upon joining a new control plane node. The corresponding encryption key is in the InitConfiguration.

LocalEtcd

Appears in:

LocalEtcd describes that kubeadm should run an etcd cluster locally

FieldDescription
ImageMeta [Required]
ImageMeta
(Members of ImageMeta are embedded into this type.)

ImageMeta allows to customize the container used for etcd.

dataDir [Required]
string

dataDir is the directory etcd will place its data. Defaults to "/var/lib/etcd".

extraArgs
map[string]string

extraArgs are extra arguments provided to the etcd binary when run inside a static Pod. A key in this map is the flag name as it appears on the command line except without leading dash(es).

serverCertSANs
[]string

serverCertSANs sets extra Subject Alternative Names (SANs) for the etcd server signing certificate.

peerCertSANs
[]string

peerCertSANs sets extra Subject Alternative Names (SANs) for the etcd peer signing certificate.

Networking

Appears in:

Networking contains elements describing cluster's networking configuration

FieldDescription
serviceSubnet
string

serviceSubnet is the subnet used by Kubernetes Services. Defaults to "10.96.0.0/12".

podSubnet
string

podSubnet is the subnet used by Pods.

dnsDomain
string

dnsDomain is the DNS domain used by Kubernetes Services. Defaults to "cluster.local".

NodeRegistrationOptions

Appears in:

NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via "kubeadm init" or "kubeadm join"

FieldDescription
name
string

name is the .metadata.name field of the Node API object that will be created in this kubeadm init or kubeadm join operation. This field is also used in the CommonName field of the kubelet's client certificate to the API server. Defaults to the hostname of the node if not provided.

criSocket
string

criSocket is used to retrieve container runtime info. This information will be annotated to the Node API object, for later re-use

taints [Required]
[]core/v1.Taint

taints specifies the taints the Node API object should be registered with. If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted with a control-plane taint for control-plane nodes. If you don't want to taint your control-plane node, set this field to an empty list, i.e. taints: [] in the YAML file. This field is solely used for Node registration.

kubeletExtraArgs
map[string]string

kubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the 'kubelet-config-1.X' ConfigMap. Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. A key in this map is the flag name as it appears on the command line except without leading dash(es).

ignorePreflightErrors
[]string

ignorePreflightErrors provides a list of pre-flight errors to be ignored when the current node is registered.

imagePullPolicy
core/v1.PullPolicy

imagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations. The value of this field must be one of "Always", "IfNotPresent" or "Never". If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host.

Patches

Appears in:

Patches contains options related to applying patches to components deployed by kubeadm.

FieldDescription
directory
string

directory is a path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd". "patchtype" can be one of "strategic" "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.

BootstrapToken

Appears in:

BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

FieldDescription
token [Required]
BootstrapTokenString

token is used for establishing bidirectional trust between nodes and control-planes. Used for joining nodes in the cluster.

description
string

description sets a human-friendly message why this token exists and what it's used for, so other administrators can know its purpose.

ttl
meta/v1.Duration

ttl defines the time to live for this token. Defaults to 24h. expires and ttl are mutually exclusive.

expires
meta/v1.Time

expires specifies the timestamp when this token expires. Defaults to being set dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

usages
[]string

usages describes the ways in which this token can be used. Can by default be used for establishing bidirectional trust, but that can be changed here.

groups
[]string

groups specifies the extra groups that this token will authenticate as when/if used for authentication

BootstrapTokenString

Appears in:

BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used for both validation of the practically of the API server from a joining node's point of view and as an authentication method for the node in the bootstrap phase of "kubeadm join". This token is and should be short-lived.

FieldDescription
- [Required]
string
No description provided.
- [Required]
string
No description provided.

14 - Kubelet Configuration (v1alpha1)

Resource Types

FormatOptions

Appears in:

FormatOptions contains options for the different logging formats.

FieldDescription
json [Required]
JSONOptions

[Experimental] JSON contains options for logging format "json".

JSONOptions

Appears in:

JSONOptions contains options for logging format "json".

FieldDescription
splitStream [Required]
bool

[Experimental] SplitStream redirects error messages to stderr while info messages go to stdout, with buffering. The default is to write both to stdout, without buffering.

infoBufferSize [Required]
k8s.io/apimachinery/pkg/api/resource.QuantityValue

[Experimental] InfoBufferSize sets the size of the info stream when using split streams. The default is zero, which disables buffering.

VModuleConfiguration

(Alias of []k8s.io/component-base/config/v1alpha1.VModuleItem)

Appears in:

VModuleConfiguration is a collection of individual file names or patterns and the corresponding verbosity threshold.

CredentialProviderConfig

CredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1alpha1
kind
string
CredentialProviderConfig
providers [Required]
[]CredentialProvider

providers is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is used.

CredentialProvider

Appears in:

CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).

FieldDescription
name [Required]
string

name is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag).

matchImages [Required]
[]string

matchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.

Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like *.k8s.io or k8s.*.io, and top-level-domains such as k8s.*. Matching partial subdomains like app*.k8s.io is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.

A match exists between an image and a matchImage when all of the below are true:

  • Both contain the same number of domain parts and each part matches.
  • The URL path of an imageMatch must be a prefix of the target image URL path.
  • If the imageMatch contains a port, then the port must match in the image as well.

Example values of matchImages:

  • 123456789.dkr.ecr.us-east-1.amazonaws.com
  • *.azurecr.io
  • gcr.io
  • *.*.registry.io
  • registry.io:8080/path
defaultCacheDuration [Required]
meta/v1.Duration

defaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.

apiVersion [Required]
string

Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Current supported values are:

  • credentialprovider.kubelet.k8s.io/v1alpha1
args
[]string

Arguments to pass to the command when executing it.

env
[]ExecEnvVar

Env defines additional environment variables to expose to the process. These are unioned with the host's environment, as well as variables client-go uses to pass argument to the plugin.

ExecEnvVar

Appears in:

ExecEnvVar is used for setting environment variables when executing an exec-based credential plugin.

FieldDescription
name [Required]
string
No description provided.
value [Required]
string
No description provided.

15 - Kubelet Configuration (v1beta1)

Resource Types

CredentialProviderConfig

CredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1beta1
kind
string
CredentialProviderConfig
providers [Required]
[]CredentialProvider

providers is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is used.

KubeletConfiguration

KubeletConfiguration contains the configuration for the Kubelet

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1beta1
kind
string
KubeletConfiguration
enableServer [Required]
bool

enableServer enables Kubelet's secured server. Note: Kubelet's insecure port is controlled by the readOnlyPort option. Default: true

staticPodPath
string

staticPodPath is the path to the directory containing local (static) pods to run, or the path to a single static pod file. Default: ""

syncFrequency
meta/v1.Duration

syncFrequency is the max period between synchronizing running containers and config. Default: "1m"

fileCheckFrequency
meta/v1.Duration

fileCheckFrequency is the duration between checking config files for new data. Default: "20s"

httpCheckFrequency
meta/v1.Duration

httpCheckFrequency is the duration between checking http for new data. Default: "20s"

staticPodURL
string

staticPodURL is the URL for accessing static pods to run. Default: ""

staticPodURLHeader
map[string][]string

staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL. Default: nil

address
string

address is the IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces). Default: "0.0.0.0"

port
int32

port is the port for the Kubelet to serve on. The port number must be between 1 and 65535, inclusive. Default: 10250

readOnlyPort
int32

readOnlyPort is the read-only port for the Kubelet to serve on with no authentication/authorization. The port number must be between 1 and 65535, inclusive. Setting this field to 0 disables the read-only service. Default: 0 (disabled)

tlsCertFile
string

tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If tlsCertFile and tlsPrivateKeyFile are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to the Kubelet's --cert-dir flag. Default: ""

tlsPrivateKeyFile
string

tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile. Default: ""

tlsCipherSuites
[]string

tlsCipherSuites is the list of allowed cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). Default: nil

tlsMinVersion
string

tlsMinVersion is the minimum TLS version supported. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). Default: ""

rotateCertificates
bool

rotateCertificates enables client certificate rotation. The Kubelet will request a new certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests. Default: false

serverTLSBootstrap
bool

serverTLSBootstrap enables server certificate bootstrap. Instead of self signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. This requires an approver to approve the certificate signing requests (CSR). The RotateKubeletServerCertificate feature must be enabled when setting this field. Default: false

authentication
KubeletAuthentication

authentication specifies how requests to the Kubelet's server are authenticated. Defaults: anonymous: enabled: false webhook: enabled: true cacheTTL: "2m"

authorization
KubeletAuthorization

authorization specifies how requests to the Kubelet's server are authorized. Defaults: mode: Webhook webhook: cacheAuthorizedTTL: "5m" cacheUnauthorizedTTL: "30s"

registryPullQPS
int32

registryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. Default: 5

registryBurst
int32

registryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. Default: 10

eventRecordQPS
int32

eventRecordQPS is the maximum event creations per second. If 0, there is no limit enforced. The value cannot be a negative number. Default: 5

eventBurst
int32

eventBurst is the maximum size of a burst of event creations, temporarily allows event creations to burst to this number, while still not exceeding eventRecordQPS. This field canot be a negative number and it is only used when eventRecordQPS > 0. Default: 10

enableDebuggingHandlers
bool

enableDebuggingHandlers enables server endpoints for log access and local running of containers and commands, including the exec, attach, logs, and portforward features. Default: true

enableContentionProfiling
bool

enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true. Default: false

healthzPort
int32

healthzPort is the port of the localhost healthz endpoint (set to 0 to disable). A valid number is between 1 and 65535. Default: 10248

healthzBindAddress
string

healthzBindAddress is the IP address for the healthz server to serve on. Default: "127.0.0.1"

oomScoreAdj
int32

oomScoreAdj is The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. Default: -999

clusterDomain
string

clusterDomain is the DNS domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains. Default: ""

clusterDNS
[]string

clusterDNS is a list of IP addresses for the cluster DNS server. If set, kubelet will configure all containers to use this for DNS resolution instead of the host's DNS servers. Default: nil

streamingConnectionIdleTimeout
meta/v1.Duration

streamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. Default: "4h"

nodeStatusUpdateFrequency
meta/v1.Duration

nodeStatusUpdateFrequency is the frequency that kubelet computes node status. If node lease feature is not enabled, it is also the frequency that kubelet posts node status to master. Note: When node lease feature is not enabled, be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. Default: "10s"

nodeStatusReportFrequency
meta/v1.Duration

nodeStatusReportFrequency is the frequency that kubelet posts node status to master if node status does not change. Kubelet will ignore this frequency and post node status immediately if any change is detected. It is only used when node lease feature is enabled. nodeStatusReportFrequency's default value is 5m. But if nodeStatusUpdateFrequency is set explicitly, nodeStatusReportFrequency's default value will be set to nodeStatusUpdateFrequency for backward compatibility. Default: "5m"

nodeLeaseDurationSeconds
int32

nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease. NodeLease provides an indicator of node health by having the Kubelet create and periodically renew a lease, named after the node, in the kube-node-lease namespace. If the lease expires, the node can be considered unhealthy. The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal interval may be set based on the lease duration. The field value must be greater than 0. Default: 40

imageMinimumGCAge
meta/v1.Duration

imageMinimumGCAge is the minimum age for an unused image before it is garbage collected. Default: "2m"

imageGCHighThresholdPercent
int32

imageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run. The percent is calculated by dividing this field value by 100, so this field must be between 0 and 100, inclusive. When specified, the value must be greater than imageGCLowThresholdPercent. Default: 85

imageGCLowThresholdPercent
int32

imageGCLowThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated by dividing this field value by 100, so the field value must be between 0 and 100, inclusive. When specified, the value must be less than imageGCHighThresholdPercent. Default: 80

volumeStatsAggPeriod
meta/v1.Duration

volumeStatsAggPeriod is the frequency for calculating and caching volume disk usage for all pods. Default: "1m"

kubeletCgroups
string

kubeletCgroups is the absolute name of cgroups to isolate the kubelet in Default: ""

systemCgroups
string

systemCgroups is absolute name of cgroups in which to place all non-kernel processes that are not already in a container. Empty for no container. Rolling back the flag requires a reboot. The cgroupRoot must be specified if this field is not empty. Default: ""

cgroupRoot
string

cgroupRoot is the root cgroup to use for pods. This is handled by the container runtime on a best effort basis.

cgroupsPerQOS
bool

cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes and all Burstable and BestEffort Pods are brought up under their specific top level QoS CGroup. Default: true

cgroupDriver
string

cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs or systemd). Default: "cgroupfs"

cpuManagerPolicy
string

cpuManagerPolicy is the name of the policy to use. Requires the CPUManager feature gate to be enabled. Default: "None"

cpuManagerPolicyOptions
map[string]string

cpuManagerPolicyOptions is a set of key=value which allows to set extra options to fine tune the behaviour of the cpu manager policies. Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled. Default: nil

cpuManagerReconcilePeriod
meta/v1.Duration

cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager. Requires the CPUManager feature gate to be enabled. Default: "10s"

memoryManagerPolicy
string

memoryManagerPolicy is the name of the policy to use by memory manager. Requires the MemoryManager feature gate to be enabled. Default: "none"

topologyManagerPolicy
string

topologyManagerPolicy is the name of the topology manager policy to use. Valid values include:

  • restricted: kubelet only allows pods with optimal NUMA node alignment for requested resources;
  • best-effort: kubelet will favor pods with NUMA alignment of CPU and device resources;
  • none: kubelet has no knowledge of NUMA alignment of a pod's CPU and device resources.
  • single-numa-node: kubelet only allows pods with a single NUMA alignment of CPU and device resources.

Policies other than "none" require the TopologyManager feature gate to be enabled. Default: "none"

topologyManagerScope
string

topologyManagerScope represents the scope of topology hint generation that topology manager requests and hint providers generate. Valid values include:

  • container: topology policy is applied on a per-container basis.
  • pod: topology policy is applied on a per-pod basis.

"pod" scope requires the TopologyManager feature gate to be enabled. Default: "container"

qosReserved
map[string]string

qosReserved is a set of resource name to percentage pairs that specify the minimum percentage of a resource reserved for exclusive use by the guaranteed QoS tier. Currently supported resources: "memory" Requires the QOSReserved feature gate to be enabled. Default: nil

runtimeRequestTimeout
meta/v1.Duration

runtimeRequestTimeout is the timeout for all runtime requests except long running requests - pull, logs, exec and attach. Default: "2m"

hairpinMode
string

hairpinMode specifies how the Kubelet should configure the container bridge for hairpin packets. Setting this flag allows endpoints in a Service to loadbalance back to themselves if they should try to access their own Service. Values:

  • "promiscuous-bridge": make the container bridge promiscuous.
  • "hairpin-veth": set the hairpin flag on container veth interfaces.
  • "none": do nothing.

Generally, one must set --hairpin-mode=hairpin-veth to achieve hairpin NAT, because promiscuous-bridge assumes the existence of a container bridge named cbr0. Default: "promiscuous-bridge"

maxPods
int32

maxPods is the maximum number of Pods that can run on this Kubelet. The value must be a non-negative integer. Default: 110

podCIDR
string

podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the control plane. Default: ""

podPidsLimit
int64

podPidsLimit is the maximum number of PIDs in any pod. Default: -1

resolvConf
string

resolvConf is the resolver configuration file used as the basis for the container DNS resolution configuration. If set to the empty string, will override the default and effectively disable DNS lookups. Default: "/etc/resolv.conf"

runOnce
bool

runOnce causes the Kubelet to check the API server once for pods, run those in addition to the pods specified by static pod files, and exit. Default: false

cpuCFSQuota
bool

cpuCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. Default: true

cpuCFSQuotaPeriod
meta/v1.Duration

cpuCFSQuotaPeriod is the CPU CFS quota period value, cpu.cfs_period_us. The value must be between 1 us and 1 second, inclusive. Requires the CustomCPUCFSQuotaPeriod feature gate to be enabled. Default: "100ms"

nodeStatusMaxImages
int32

nodeStatusMaxImages caps the number of images reported in Node.status.images. The value must be greater than -2. Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned. Default: 50

maxOpenFiles
int64

maxOpenFiles is Number of files that can be opened by Kubelet process. The value must be a non-negative number. Default: 1000000

contentType
string

contentType is contentType of requests sent to apiserver. Default: "application/vnd.kubernetes.protobuf"

kubeAPIQPS
int32

kubeAPIQPS is the QPS to use while talking with kubernetes apiserver. Default: 5

kubeAPIBurst
int32

kubeAPIBurst is the burst to allow while talking with kubernetes API server. This field cannot be a negative number. Default: 10

serializeImagePulls
bool

serializeImagePulls when enabled, tells the Kubelet to pull images one at a time. We recommend not changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. Default: true

evictionHard
map[string]string

evictionHard is a map of signal names to quantities that defines hard eviction thresholds. For example: {"memory.available": "300Mi"}. To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource. Default: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%"

evictionSoft
map[string]string

evictionSoft is a map of signal names to quantities that defines soft eviction thresholds. For example: {"memory.available": "300Mi"}. Default: nil

evictionSoftGracePeriod
map[string]string

evictionSoftGracePeriod is a map of signal names to quantities that defines grace periods for each soft eviction signal. For example: {"memory.available": "30s"}. Default: nil

evictionPressureTransitionPeriod
meta/v1.Duration

evictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. Default: "5m"

evictionMaxPodGracePeriod
int32

evictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. This value effectively caps the Pod's terminationGracePeriodSeconds value during soft evictions. Note: Due to issue #64530, the behavior has a bug where this value currently just overrides the grace period during soft eviction, which can increase the grace period from what is set on the Pod. This bug will be fixed in a future release. Default: 0

evictionMinimumReclaim
map[string]string

evictionMinimumReclaim is a map of signal names to quantities that defines minimum reclaims, which describe the minimum amount of a given resource the kubelet will reclaim when performing a pod eviction while that resource is under pressure. For example: {"imagefs.available": "2Gi"}. Default: nil

podsPerCore
int32

podsPerCore is the maximum number of pods per core. Cannot exceed maxPods. The value must be a non-negative integer. If 0, there is no limit on the number of Pods. Default: 0

enableControllerAttachDetach
bool

enableControllerAttachDetach enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. Note: attaching/detaching CSI volumes is not supported by the kubelet, so this option needs to be true for that use case. Default: true

protectKernelDefaults
bool

protectKernelDefaults, if true, causes the Kubelet to error if kernel flags are not as it expects. Otherwise the Kubelet will attempt to modify kernel flags to match its expectation. Default: false

makeIPTablesUtilChains
bool

makeIPTablesUtilChains, if true, causes the Kubelet ensures a set of iptables rules are present on host. These rules will serve as utility rules for various components, e.g. kube-proxy. The rules will be created based on iptablesMasqueradeBit and iptablesDropBit. Default: true

iptablesMasqueradeBit
int32

iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT. Values must be within the range [0, 31]. Must be different from other mark bits. Warning: Please match the value of the corresponding parameter in kube-proxy. TODO: clean up IPTablesMasqueradeBit in kube-proxy. Default: 14

iptablesDropBit
int32

iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets. Values must be within the range [0, 31]. Must be different from other mark bits. Default: 15

featureGates
map[string]bool

featureGates is a map of feature names to bools that enable or disable experimental features. This field modifies piecemeal the built-in default values from "k8s.io/kubernetes/pkg/features/kube_features.go". Default: nil

failSwapOn
bool

failSwapOn tells the Kubelet to fail to start if swap is enabled on the node. Default: true

memorySwap
MemorySwapConfiguration

memorySwap configures swap memory available to container workloads.

containerLogMaxSize
string

containerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". Default: "10Mi"

containerLogMaxFiles
int32

containerLogMaxFiles specifies the maximum number of container log files that can be present for a container. Default: 5

configMapAndSecretChangeDetectionStrategy
ResourceChangeDetectionStrategy

configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret managers are running. Valid values include:

  • Get: kubelet fetches necessary objects directly from the API server;
  • Cache: kubelet uses TTL cache for object fetched from the API server;
  • Watch: kubelet uses watches to observe changes to objects that are in its interest.

Default: "Watch"

systemReserved
map[string]string

systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. Default: nil

kubeReserved
map[string]string

kubeReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local storage for root file system are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more details. Default: nil

reservedSystemCPUs [Required]
string

The reservedSystemCPUs option specifies the CPU list reserved for the host level system threads and kubernetes related threads. This provide a "static" CPU list rather than the "dynamic" list by systemReserved and kubeReserved. This option does not support systemReservedCgroup or kubeReservedCgroup.

showHiddenMetricsForVersion
string

showHiddenMetricsForVersion is the previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: 1.16. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that. Default: ""

systemReservedCgroup
string

systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used to enforce systemReserved compute resource reservation for OS system daemons. Refer to Node Allocatable doc for more information. Default: ""

kubeReservedCgroup
string

kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used to enforce KubeReserved compute resource reservation for Kubernetes node system daemons. Refer to Node Allocatable doc for more information. Default: ""

enforceNodeAllocatable
[]string

This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform. This flag accepts a list of options. Acceptable options are none, pods, system-reserved and kube-reserved. If none is specified, no other options may be specified. When system-reserved is in the list, systemReservedCgroup must be specified. When kube-reserved is in the list, kubeReservedCgroup must be specified. This field is supported only when cgroupsPerQOS is set to true. Refer to Node Allocatable for more information. Default: ["pods"]

allowedUnsafeSysctls
[]string

A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in *). Unsafe sysctl groups are kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue.*, and net.*. For example: "kernel.msg*,net.ipv4.route.min_pmtu" Default: []

volumePluginDir
string

volumePluginDir is the full path of the directory in which to search for additional third party volume plugins. Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"

providerID
string

providerID, if set, sets the unique ID of the instance that an external provider (i.e. cloudprovider) can use to identify a specific node. Default: ""

kernelMemcgNotification
bool

kernelMemcgNotification, if set, instructs the the kubelet to integrate with the kernel memcg notification for determining if memory eviction thresholds are exceeded rather than polling. Default: false

logging [Required]
LoggingConfiguration

logging specifies the options of logging. Refer to Logs Options for more information. Default: Format: text

enableSystemLogHandler
bool

enableSystemLogHandler enables system logs via web interface host:port/logs/ Default: true

shutdownGracePeriod
meta/v1.Duration

shutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown. Default: "0s"

shutdownGracePeriodCriticalPods
meta/v1.Duration

shutdownGracePeriodCriticalPods specifies the duration used to terminate critical pods during a node shutdown. This should be less than shutdownGracePeriod. For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s, during a node shutdown the first 20 seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods. Default: "0s"

shutdownGracePeriodByPodPriority
[]ShutdownGracePeriodByPodPriority

shutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based on their associated priority class value. When a shutdown request is received, the Kubelet will initiate shutdown on all pods running on the node with a grace period that depends on the priority of the pod, and then wait for all pods to exit. Each entry in the array represents the graceful shutdown time a pod with a priority class value that lies in the range of that value and the next higher entry in the list when the node is shutting down. For example, to allow critical pods 10s to shutdown, priority>=10000 pods 20s to shutdown, and all remaining pods 30s to shutdown.

shutdownGracePeriodByPodPriority:

  • priority: 2000000000 shutdownGracePeriodSeconds: 10
  • priority: 10000 shutdownGracePeriodSeconds: 20
  • priority: 0 shutdownGracePeriodSeconds: 30

The time the Kubelet will wait before exiting will at most be the maximum of all shutdownGracePeriodSeconds for each priority class range represented on the node. When all pods have exited or reached their grace periods, the Kubelet will release the shutdown inhibit lock. Requires the GracefulNodeShutdown feature gate to be enabled. This configuration must be empty if either ShutdownGracePeriod or ShutdownGracePeriodCriticalPods is set. Default: nil

reservedMemory
[]MemoryReservation

reservedMemory specifies a comma-separated list of memory reservations for NUMA nodes. The parameter makes sense only in the context of the memory manager feature. The memory manager will not allocate reserved memory for container workloads. For example, if you have a NUMA0 with 10Gi of memory and the reservedMemory was specified to reserve 1Gi of memory at NUMA0, the memory manager will assume that only 9Gi is available for allocation. You can specify a different amount of NUMA node and memory types. You can omit this parameter at all, but you should be aware that the amount of reserved memory from all NUMA nodes should be equal to the amount of memory specified by the node allocatable. If at least one node allocatable parameter has a non-zero value, you will need to specify at least one NUMA node. Also, avoid specifying:

  1. Duplicates, the same NUMA node, and memory type, but with a different value.
  2. zero limits for any memory type.
  3. NUMAs nodes IDs that do not exist under the machine.
  4. memory types except for memory and hugepages-

Default: nil

enableProfilingHandler
bool

enableProfilingHandler enables profiling via web interface host:port/debug/pprof/ Default: true

enableDebugFlagsHandler
bool

enableDebugFlagsHandler enables flags endpoint via web interface host:port/debug/flags/v Default: true

seccompDefault
bool

SeccompDefault enables the use of RuntimeDefault as the default seccomp profile for all workloads. This requires the corresponding SeccompDefault feature gate to be enabled as well. Default: false

memoryThrottlingFactor
float64

MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory when setting the cgroupv2 memory.high value to enforce MemoryQoS. Decreasing this factor will set lower high limit for container cgroups and put heavier reclaim pressure while increasing will put less reclaim pressure. See http://kep.k8s.io/2570 for more details. Default: 0.8

registerWithTaints
[]core/v1.Taint

registerWithTaints are an array of taints to add to a node object when the kubelet registers itself. This only takes effect when registerNode is true and upon the initial registration of the node. Default: nil

registerNode
bool

registerNode enables automatic registration with the apiserver. Default: true

SerializedNodeConfigSource

SerializedNodeConfigSource allows us to serialize v1.NodeConfigSource. This type is used internally by the Kubelet for tracking checkpointed dynamic configs. It exists in the kubeletconfig API group because it is classified as a versioned input to the Kubelet.

FieldDescription
apiVersion
string
kubelet.config.k8s.io/v1beta1
kind
string
SerializedNodeConfigSource
source
core/v1.NodeConfigSource

source is the source that we are serializing.

CredentialProvider

Appears in:

CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).

FieldDescription
name [Required]
string

name is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag).

matchImages [Required]
[]string

matchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.

Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.

A match exists between an image and a matchImage when all of the below are true:

  • Both contain the same number of domain parts and each part matches.
  • The URL path of an imageMatch must be a prefix of the target image URL path.
  • If the imageMatch contains a port, then the port must match in the image as well.

Example values of matchImages:

  • 123456789.dkr.ecr.us-east-1.amazonaws.com
  • *.azurecr.io
  • gcr.io
  • ..registry.io
  • registry.io:8080/path
defaultCacheDuration [Required]
meta/v1.Duration

defaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.

apiVersion [Required]
string

Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Current supported values are:

  • credentialprovider.kubelet.k8s.io/v1beta1
args
[]string

Arguments to pass to the command when executing it.

env
[]ExecEnvVar

Env defines additional environment variables to expose to the process. These are unioned with the host's environment, as well as variables client-go uses to pass argument to the plugin.

ExecEnvVar

Appears in:

ExecEnvVar is used for setting environment variables when executing an exec-based credential plugin.

FieldDescription
name [Required]
string
No description provided.
value [Required]
string
No description provided.

KubeletAnonymousAuthentication

Appears in:

FieldDescription
enabled
bool

enabled allows anonymous requests to the kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.

KubeletAuthentication

Appears in:

FieldDescription
x509
KubeletX509Authentication

x509 contains settings related to x509 client certificate authentication.

webhook
KubeletWebhookAuthentication

webhook contains settings related to webhook bearer token authentication.

anonymous
KubeletAnonymousAuthentication

anonymous contains settings related to anonymous authentication.

KubeletAuthorization

Appears in:

FieldDescription
mode
KubeletAuthorizationMode

mode is the authorization mode to apply to requests to the kubelet server. Valid values are AlwaysAllow and Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization.

webhook
KubeletWebhookAuthorization

webhook contains settings related to Webhook authorization.

KubeletAuthorizationMode

(Alias of string)

Appears in:

KubeletWebhookAuthentication

Appears in:

FieldDescription
enabled
bool

enabled allows bearer token authentication backed by the tokenreviews.authentication.k8s.io API.

cacheTTL
meta/v1.Duration

cacheTTL enables caching of authentication results

KubeletWebhookAuthorization

Appears in:

FieldDescription
cacheAuthorizedTTL
meta/v1.Duration

cacheAuthorizedTTL is the duration to cache 'authorized' responses from the webhook authorizer.

cacheUnauthorizedTTL
meta/v1.Duration

cacheUnauthorizedTTL is the duration to cache 'unauthorized' responses from the webhook authorizer.

KubeletX509Authentication

Appears in:

FieldDescription
clientCAFile
string

clientCAFile is the path to a PEM-encoded certificate bundle. If set, any request presenting a client certificate signed by one of the authorities in the bundle is authenticated with a username corresponding to the CommonName, and groups corresponding to the Organization in the client certificate.

MemoryReservation

Appears in:

MemoryReservation specifies the memory reservation of different types for each NUMA node

FieldDescription
numaNode [Required]
int32
No description provided.
limits [Required]
core/v1.ResourceList
No description provided.

MemorySwapConfiguration

Appears in:

FieldDescription
swapBehavior
string

swapBehavior configures swap memory available to container workloads. May be one of "", "LimitedSwap": workload combined memory and swap usage cannot exceed pod memory limit "UnlimitedSwap": workloads can use unlimited swap, up to the allocatable limit.

ResourceChangeDetectionStrategy

(Alias of string)

Appears in:

ResourceChangeDetectionStrategy denotes a mode in which internal managers (secret, configmap) are discovering object changes.

ShutdownGracePeriodByPodPriority

Appears in:

ShutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based on their associated priority class value

FieldDescription
priority [Required]
int32

priority is the priority value associated with the shutdown grace period

shutdownGracePeriodSeconds [Required]
int64

shutdownGracePeriodSeconds is the shutdown grace period in seconds

FormatOptions

Appears in:

FormatOptions contains options for the different logging formats.

FieldDescription
json [Required]
JSONOptions

[Experimental] JSON contains options for logging format "json".

JSONOptions

Appears in:

JSONOptions contains options for logging format "json".

FieldDescription
splitStream [Required]
bool

[Experimental] SplitStream redirects error messages to stderr while info messages go to stdout, with buffering. The default is to write both to stdout, without buffering.

infoBufferSize [Required]
k8s.io/apimachinery/pkg/api/resource.QuantityValue

[Experimental] InfoBufferSize sets the size of the info stream when using split streams. The default is zero, which disables buffering.

LoggingConfiguration

Appears in:

LoggingConfiguration contains logging options Refer Logs Options for more information.

FieldDescription
format [Required]
string

Format Flag specifies the structure of log messages. default value of format is text

flushFrequency [Required]
time.Duration

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log flushes. Ignored if the selected logging backend writes log messages without buffering.

verbosity [Required]
uint32

Verbosity is the threshold that determines which log messages are logged. Default is zero which logs only the most important messages. Higher values enable additional messages. Error messages are always logged.

vmodule [Required]
VModuleConfiguration

VModule overrides the verbosity threshold for individual files. Only supported for "text" log format.

options [Required]
FormatOptions

[Experimental] Options holds additional parameters that are specific to the different logging formats. Only the options for the selected format get used, but all of them get validated.

VModuleConfiguration

(Alias of []k8s.io/component-base/config/v1alpha1.VModuleItem)

Appears in:

VModuleConfiguration is a collection of individual file names or patterns and the corresponding verbosity threshold.

16 - Kubelet CredentialProvider (v1alpha1)

Resource Types

CredentialProviderRequest

CredentialProviderRequest includes the image that the kubelet requires authentication for. Kubelet will pass this request object to the plugin via stdin. In general, plugins should prefer responding with the same apiVersion they were sent.

FieldDescription
apiVersion
string
credentialprovider.kubelet.k8s.io/v1alpha1
kind
string
CredentialProviderRequest
image [Required]
string

image is the container image that is being pulled as part of the credential provider plugin request. Plugins may optionally parse the image to extract any information required to fetch credentials.

CredentialProviderResponse

CredentialProviderResponse holds credentials that the kubelet should use for the specified image provided in the original request. Kubelet will read the response from the plugin via stdout. This response should be set to the same apiVersion as CredentialProviderRequest.

FieldDescription
apiVersion
string
credentialprovider.kubelet.k8s.io/v1alpha1
kind
string
CredentialProviderResponse
cacheKeyType [Required]
PluginCacheKeyType

cacheKeyType indiciates the type of caching key to use based on the image provided in the request. There are three valid values for the cache key type: Image, Registry, and Global. If an invalid value is specified, the response will NOT be used by the kubelet.

cacheDuration
meta/v1.Duration

cacheDuration indicates the duration the provided credentials should be cached for. The kubelet will use this field to set the in-memory cache duration for credentials in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

auth
map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1alpha1.AuthConfig

auth is a map containing authentication information passed into the kubelet. Each key is a match image string (more on this below). The corresponding authConfig value should be valid for all images that match against this key. A plugin should set this field to null if no valid credentials can be returned for the requested image.

Each key in the map is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.

The kubelet will match images against the key when all of the below are true:

  • Both contain the same number of domain parts and each part matches.
  • The URL path of an imageMatch must be a prefix of the target image URL path.
  • If the imageMatch contains a port, then the port must match in the image as well.

When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

  • longer keys come before shorter keys with the same prefix
  • non-wildcard keys come before wildcard keys with the same prefix.

For any given match, the kubelet will attempt an image pull with the provided credentials, stopping after the first successfully authenticated pull.

Example keys:

  • 123456789.dkr.ecr.us-east-1.amazonaws.com
  • *.azurecr.io
  • gcr.io
  • ..registry.io
  • registry.io:8080/path

AuthConfig

Appears in:

AuthConfig contains authentication information for a container registry. Only username/password based authentication is supported today, but more authentication mechanisms may be added in the future.

FieldDescription
username [Required]
string

username is the username used for authenticating to the container registry An empty username is valid.

password [Required]
string

password is the password used for authenticating to the container registry An empty password is valid.

PluginCacheKeyType

(Alias of string)

Appears in:

17 - Kubelet CredentialProvider (v1beta1)

Resource Types

CredentialProviderRequest

CredentialProviderRequest includes the image that the kubelet requires authentication for. Kubelet will pass this request object to the plugin via stdin. In general, plugins should prefer responding with the same apiVersion they were sent.

FieldDescription
apiVersion
string
credentialprovider.kubelet.k8s.io/v1beta1
kind
string
CredentialProviderRequest
image [Required]
string

image is the container image that is being pulled as part of the credential provider plugin request. Plugins may optionally parse the image to extract any information required to fetch credentials.

CredentialProviderResponse

CredentialProviderResponse holds credentials that the kubelet should use for the specified image provided in the original request. Kubelet will read the response from the plugin via stdout. This response should be set to the same apiVersion as CredentialProviderRequest.

FieldDescription
apiVersion
string
credentialprovider.kubelet.k8s.io/v1beta1
kind
string
CredentialProviderResponse
cacheKeyType [Required]
PluginCacheKeyType

cacheKeyType indiciates the type of caching key to use based on the image provided in the request. There are three valid values for the cache key type: Image, Registry, and Global. If an invalid value is specified, the response will NOT be used by the kubelet.

cacheDuration
meta/v1.Duration

cacheDuration indicates the duration the provided credentials should be cached for. The kubelet will use this field to set the in-memory cache duration for credentials in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

auth
map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1beta1.AuthConfig

auth is a map containing authentication information passed into the kubelet. Each key is a match image string (more on this below). The corresponding authConfig value should be valid for all images that match against this key. A plugin should set this field to null if no valid credentials can be returned for the requested image.

Each key in the map is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.

The kubelet will match images against the key when all of the below are true:

  • Both contain the same number of domain parts and each part matches.
  • The URL path of an imageMatch must be a prefix of the target image URL path.
  • If the imageMatch contains a port, then the port must match in the image as well.

When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

  • longer keys come before shorter keys with the same prefix
  • non-wildcard keys come before wildcard keys with the same prefix.

For any given match, the kubelet will attempt an image pull with the provided credentials, stopping after the first successfully authenticated pull.

Example keys:

  • 123456789.dkr.ecr.us-east-1.amazonaws.com
  • *.azurecr.io
  • gcr.io
  • ..registry.io
  • registry.io:8080/path

AuthConfig

Appears in:

AuthConfig contains authentication information for a container registry. Only username/password based authentication is supported today, but more authentication mechanisms may be added in the future.

FieldDescription
username [Required]
string

username is the username used for authenticating to the container registry An empty username is valid.

password [Required]
string

password is the password used for authenticating to the container registry An empty password is valid.

PluginCacheKeyType

(Alias of string)

Appears in:

18 - WebhookAdmission Configuration (v1)

Package v1 is the v1 version of the API.

Resource Types

WebhookAdmission

WebhookAdmission provides configuration for the webhook admission controller.

FieldDescription
apiVersion
string
apiserver.config.k8s.io/v1
kind
string
WebhookAdmission
kubeConfigFile [Required]
string

KubeConfigFile is the path to the kubeconfig file.