path: root/doc/rgmanager-pacemaker.resources.txt

CLUSTERED RESOURCE PROPERTIES IN THE LIGHT OF RGMANAGER-PACEMAKER CONVERSION

Copyright 2014 Red Hat, Inc., Jan Pokorný <jpokorny @at@ Red Hat .dot. com>
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".


Intro
=====

This document elaborates on how selected resource relationship properties
(denoting the run-time behavior) formalized by the means of LTL logic maps
to particular RGManager (R) and Pacemaker (P) configuration arrangements.
Due to the purpose of this document, "selected" here means set of
properties one commonly uses in case of the former cluster resource
manager (R).

Properties are categorised, each is further dissected based on
the property variants (basically holds or doesn't, but can be more
convoluted), and for each variants, the LTL model and R+P specifics
are provided.

Outline
-------

Resource-resource interaction properites, PROPERTY(RESOURCE1, RESOURCE2):
- ORDERING, COOCCURENCE
Absolute resource-node assignment properties, PROPERTY(RESOURCE, NODE):
- AFFINITY
Relative resource-node assignment properties, PROPERTY(RESOURCE):
- STICKY, EXCLUSIVE
Other resource properties, PROPERTY(RESOURCE):
- MANAGED

Prerequisities
--------------

Optionally, basic knowledge of LTL logic [1].
The meaning of symbols used (mind the ASCII range) goes, ordered by
descending precedence priority, like this:

. a-z          ... booleans representing satisfaction of the connected claim
. ()           ... braces (changing evaluation order of enclosed expression)
. union        ... set union, written as a function for 3+ sets
. intersection ... set intersection, written as a function for 3+ sets
. \            ... set difference
. in           ... set's item selector
.  ~           ... negation
. X,G,F,U,R    ... temporal operators (LTL)
. AND          ... conjuction
. OR           ... disjunction
. exists       ... existential quantifier (predicate logic)
. for all      ... universal quantifier (predicate logic)
. ->           ... implication (--> as "maps to" in function signature context)

There are also following sets assumed:

{}          ... empty set
2^Z         ... potential set of a set denoted with Z
NODES       ... set of all nodes
RESOURCES   ... set of all resources/services

and these functions:

RUNNABLE: NODES --> RESOURCES
... all resources that can run on given node
SCORE:    RESOURCES x NODES --> {0, 1, ...}
... order of preference for given resource to run on given node (without
    contribution of preference implied by the examined property)
ALTER(ARGS)
... alteration of the cluster behavior wrt. arguments
intersection, union
... see above
max
... given set of values, return maximum

and these predicates:

ACTIVE(A)     ... node A is active cluster member
RUNNING(A, B) ... node A runs resource B (assumes B in RUNNABLE(A))

and this oxymoron:

exists A1, A2 in NODES: A1 != A2, B in RESOURCES:
  RUNNING(A1, B) AND RUNNING(A1, B)
[given unique resource is expected to run on atmost a single node,
 we don't consider Pacemaker's clones here at all]

Notes
-----

- discreteness of the events in the LTL models is chosen quite deliberately,
  per common sense and "best fit", for the sake of simplicity
  (author is by no means expert in this field)


Resource-resource interaction properites
========================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE1, RESOURCE2),
implying modification of the behavior of cluster wrt. resource-resource
pair:

PROPERTY(RESOURCE1, RESOURCE2) -> ALTER((RESOURCE1, RESOURCE2))

Ordering (time-based interaction dependence of the resources/their states)
--------------------------------------------------------------------------

ORDERING ::= ORDERING(RESOURCE1, RESOURCE2, NONE)
           | ORDERING(RESOURCE1, RESOURCE2, WEAK)
           | ORDERING(RESOURCE1, RESOURCE2, STRONG)
           | ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)

. ORDERING(RESOURCE1, RESOURCE2, NONE)   ... no time-based dependency
                                             of the states
. ORDERING(RESOURCE1, RESOURCE2, WEAK)   ... start of 2nd preceeded by start
                                             of 1st (stop viceversa/LIFO),
                                             but only when both events happen
                                             at the same time
. ORDERING(RESOURCE1, RESOURCE2, STRONG) ... runtime of 2nd won't exceed
                                             runtime of 1st
. ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)
                                         ... runtime of 2nd won't exceed
                                             runtime of 1st (as STRONG) only
                                             at the start phase (XXX then
                                             the lifetimes are not
                                             correlated???)

A1, A2 in NODES
B in RUNNABLE(A1), C in RUNNABLE(A2)

[2. weak ordering]
ORDERING(B, C, WEAK)
~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> (X a OR TRUE) AND X X b  [start: B, then C]
a AND b AND ~c AND ~d -> X ~b AND X X ~a          [stop:  C, then B]

[3. strong ordering]
ORDERING(B, C, STRONG)
~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]
a AND b AND ~c AND ~d -> X ~b AND X X ~a  [see weak ordering]
a AND b AND ~c -> a AND b AND ~c AND ~d   [stronger, follows as per previous]

[4. asymmetric ordering]
ORDERING(B, C, ASYMMETRIC)
~~~~~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]

. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... intention RUNNING(A1, B)
. d ... intention RUNNING(A2, C)

R:
XXX implicit and explicit ordering


Cooccurence (location-based interaction dependence of the resources)
--------------------------------------------------------------------

COOCCURENCE ::= COOCURENCE(RESOURCE1, RESOURCE2, NONE)
              | COOCURENCE(RESOURCE1, RESOURCE2, POSITIVE)
              | COOCURENCE(RESOURCE1, RESOURCE2, NEGATIVE)
              | COOCURENCE(RESOURCE1, RESOURCE2, SCORE),
                SCORE in {..., -1, 0, 1, ...}, 0~NONE, -INF~NEGATIVE,
                                               +INF~POSITIVE

. COOCCURENCE(RESOURCE1, RESOURCE2, NONE)     ... not any occurence
                                                  relationship (default)
. COOCCURENCE(RESOURCE1, RESOURCE2, POSITIVE) ... positive occurence
                                                  relationship (flat model)
. COOCCURENCE(RESOURCE1, RESOURCE2, NEGATIVE) ... negative occurence
                                                  relationship (flat model)
. COOCCURENCE(RESOURCE1, RESOURCE2, SCORE)    ... score-based/advisory
                                                  occurence relationship

note: COOCCURENCE relation between RESOURCE1 and RESOURCE2 is not symmetric,
      RESOURCE1 is "dependent", RESOURCE2 is "leader"

[2. possitive cooccurence]
A1, A2 in NODES
B, C in RUNNABLE(A1) intersection RUNNABLE(A2)
COOCCURENCE(B, C, POSITIVE)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> c  [basic positive cooccurence condition]
X b -> X d    ["dependent" follows its "leader"]
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... A1 == A2
. d ... RUNNING(A2, B)

R: driven by putting set of sequentially dependent resources hierarchically
   (subsequent nesting) into service/resource group
   - __independent_subtree must not be used

P: driven by colocation constraint
   - using INFINITY as value

[3. negative cooccurence]
A1, A2 in NODES
B, C in RUNNABLE(A1) intersection RUNNABLE(A2)
COOCCURENCE(B, C, NEGATIVE)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> ~c  [basic negative cooccurence condition]
X b -> X ~d    ["dependent" escapes its "leader"]
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... A1 == A2
. d ... RUNNING(A2, B)

R: driven/emulated solely by disjunct failover domains
   - XXX and possibly with follow_service.sl in central_processing mode

P: driven by colocation constraint
   - using -INFINITY as value

[4. score-based/advisory occurence]
TBD

Explicit resource-node(s) assignment property
=============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE, NODE),
implying modification of the behavior of cluster wrt. resource-node
pair:

PROPERTY(RESOURCE, NODE) -> ALTER((RESOURCE, NODE))

Resource-node affinity
----------------------

AFFINITY ::= AFFINITY(RESOURCE, NODE, NONE)
           | AFFINITY(RESOURCE, NODE, FALSE)
           | AFFINITY(RESOURCE, NODE, TRUE)
           | AFFINITY(RESOURCE, NODE, WEIGHT), WEIGHT in {..., -1, 0, 1, ...},
                                               0~NONE, -INF~FALSE, +INF~TRUE
. AFFINITY(RESOURCE, NODE, NONE)   ... no special affinity (default)
. AFFINITY(RESOURCE, NODE, FALSE)  ... anti-affinity (the node cannot run)
. AFFINITY(RESOURCE, NODE, TRUE)   ... node forms a set of executive nodes
. AFFINITY(RESOURCE, NODE, WEIGHT) ... prioritized model

[1. model of no resource-node affinity]
P: driven by location constraint (if at all as this is a default)
   - pcs: AFFINITY(RESOURCE, NODE, NONE)
     # pcs constraint location RESOURCE prefers NODE= (??? to remove it as such)
     # pcs constraint location RESOURCE prefers NODE=0
     # pcs contraint location add SOMEID RESOURCE NODE 0

[2. model of antagonist (negative) resource-node affinity]
AN in NODES^2\{}
B in RESOURCES
for all A in AN: AFFINITY(B, A, FALSE)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X ~b

R: driven by by failover domain for resource group
   - nodes NODES\AN form selected failover domain, which has to be restricted
     to prevent slipping onto any node in AN

P: driven by location constraint
   - pcs/crm_resource: AFFINITY(RESOURCE, NODE, FALSE)
     # pcs constraint location RESOURCE avoids NODE
     # pcs constraint location RESOURCE avoids NODE=INFINITY
     # pcs constraint location RESOURCE prefers NODE=-INFINITY (???)
     # pcs contraint location add SOMEID RESOURCE NODE -INFINITY
     # crm_resource --ban --resource RESOUCE --host HOST ...

[3. model of (positive) resource-node affinity]
AN in NODES^2\{}
B in intersection( { RUNNABLE(A) | A in AN } )
for all A in AN: AFFINITY(B, A, TRUE)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X b

R: driven by by failover domain for resource group
   - nodes AN form selected failover domain

P: driven by location constraint
   - pcs: AFFINITY(RESOURCE, NODE, TRUE)
     # pcs constraint location RESOURCE prefers NODE
     # pcs constraint location RESOURCE prefers NODE=INFINITY
     # pcs constraint location RESOURCE avoids NODE=-INFINITY (???)
     # pcs contraint location add SOMEID RESOURCE NODE INFINITY

[4. model of resource-node affinity with priorities]
AN in NODES^2\{}
B in intersection( { RUNNABLE(A) | A in AN } )
for all A in AN: AFFINITY(B, A, WEIGHT(B,A) )
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a AND c -> X d  [when not sticky, resource will run on the node
                   this resource has greatest affinity to]
. a ... RUNNING(A', B)
. b ... A' in AN
. c ... STICKY(B, FALSE)  <-- see below
. d ... A' in {A'' | A'' in AN: WEIGHT(B, A'') == max({WEIGHT(B, A''')
                                                       | A''' in AN})}

R: driven by by failover domain for resource group
   - nodes AN (subset with positive WEIGHT) form selected failover domain,
     which has to be ordered, each failoverdomain prioritized as per
     respective WEIGHTs

P: driven by location constraint
   - pcs: AFFINITY(RESOURCE, NODE, WEIGHT)
     # pcs constraint location RESOURCE {prefers|avoids} NODE=WEIGHT
     # pcs contraint location add SOMEID RESOURCE NODE WEIGHT


Relative resource-node assignment properties
============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE),
implying modification of the behavior of cluster wrt. the node
running RESOURCE:

PROPERTY(RESOURCE) AND RUNNING(NODE, RESOURCE) -> ALTER(NODE)

Resource stickiness property (not moving back to preferred location)
--------------------------------------------------------------------

STICKY ::= STICKY(RESOURCE, FALSE)
         | STICKY(RESOURCE, TRUE)
         | STICKY(RESOURCE, STICKINESS), STICKINESS in {0, 1, 2, ...},
                                         0~FALSE

. STICKY(RESOURCE, FALSE)      ... unsticky/cruising resource (default)
. STICKY(RESOURCE, TRUE)       ... sticky resource (flat model)
. STICKY(RESOURCE, STICKINESS) ... sticky resource (prioritized model)

A1, A2 in NODES
B in RUNNABLE(A1) AND B in RUNNABLE(A2)
SCORE(B, A1) < SCORE(B, A2)

[1. model of unsticky/cruising resource]
STICKY(RESOURCE, FALSE)
~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X c

[2. model of sticky resource]
STICKY(RESOURCE, TRUE)
~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X a
. a ... RUNNING(A1, B)
. b ... ACTIVE(A2)
. c ... RUNNING(A2, B) resource X running (relocation if A1 != A2)

[3. model of sticky resource with priorities]
STICKY(RESOURCE, STICKINESS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TBD

R: driven by /cluster/rm/failoverdomains/failoverdomain/@nofailback
   - note: only applies to service/vm (not primitive resources)
   - 1. unsticky resource model
     - default (@nofailback=0)
   - 2. sticky resource model
     - when @nofailback is a positive number
   - 3. XXX not supported unless a way to emulate this
        using ordered failover domains(?)

P: driven by stickiness parameter
   - 1. unsticky resource model
     - default (stickiness = 0)
   - 2.,3. sticky resource model (with priorities)
     - when stickiness specified as positive number
     - group is a sum of stickiness values of underlying resources


Node-exclusiveness resource property (optionally with priority-based preemption)
--------------------------------------------------------------------------------

EXCLUSIVE ::= EXCLUSIVE(RESOURCE, FALSE)
            | EXCLUSIVE(RESOURCE, TRUE)
            | EXCLUSIVE(RESOURCE, PRIORITY), PRIORITY in {0, 1, ...}, 0~FALSE
. EXCLUSIVE(RESOURCE, FALSE)    ... non-exclusive resource (default)
. EXCLUSIVE(RESOURCE, TRUE)     ... exclusive (flat model)
. EXCLUSIVE(RESOURCE, PRIORITY) ... exclusive (prioritized pre-emptive model)

[1. model of non-node-exclusive/co-occurence-positive resource, based on 2.]
A in NODES
B in RUNNABLE(A)
for all B' in RUNNABLE(A): EXCLUSIVE(B', FALSE) [implies EXCLUSIVE(B, FALSE)]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~(a OR b) OR ~(~a -> ~b R ~a AND ~b -> ~a R ~b)
= ~(a OR b) OR ~(~a -> ~b R ~a) OR ~(~b -> ~a R ~b)
= ~(a OR b) OR ~(a OR ~b R ~a) OR ~(b OR ~a R ~b)
= ~(a OR b) OR ~a AND ~(~b R ~a) OR ~b AND ~(~a R ~b)
= ~(a OR b) OR ~a AND ~(~(a U b)) OR ~b AND ~(~(b U a))
= ~(a OR b) OR ~a AND a U b OR ~b AND b U a
= ~(a OR b) OR b OR a
= ~a AND ~b OR b OR a [= true, i.e. no restriction wrt. modelled property, QED]
. a ... RUNNING(A, B)
. b ... exists B' in 2^RUNNABLE(A)\{}: RUNNING(A, B')

[2. flat model of node-exclusive/co-occurence-less resource, no preemption]
A in NODES
B in RUNNABLE(A)
EXCLUSIVE(B, TRUE)
~~~~~~~~~~~~~~~~~~
~(a OR b) OR (~a -> ~b R ~a) AND (~b -> ~a R ~b)  [mutual exclusion]
. a ... RUNNING(A, B)
. b ... exists YS in 2^(RUNNABLE(A)\{B})\{}: for all Y in YS: RUNNING(A, Y)
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

[3. model of prioritized pre-emptive node-exclusive resource]
A in NODES
B, C in RUNNABLE(A)
I, J in {0, 1, ...}: I > J
EXCLUSIVE(B, I)
EXCLUSIVE(C, J)
~~~~~~~~~~~~~~~~~~~~~~~~~~
~(a OR c) OR (~a -> ~c R ~a) AND (~c -> ~a R ~c),  [mutual exclusion]
(b AND X a) -> (X ~b)                              [exlusivness priority wins]
. a ... RUNNING(A, B)
. b ... RUNNING(C, B)
. c ... exists YS in 2^({Y | Y in RUNNABLE(A)
                             AND (EXCLUSIVE(Y, FALSE)
                                  OR exists K > I: EXCLUSIVE(Y, K))}\{B}
                        )\{}: for all Y' in YS: RUNNING(A, Y')
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

R: driven by /cluster/rm/<service>/@exclusive
   - note: only applies to service/vm (not primitive resources)
   - 1. no node-exclusiveness model
     - default (@exclusive = 0)
   - 2. flat model
     - by default (not in central_processing mode)
   - 3. pre-emptive model
     - only in central_processing mode
     - https://access.redhat.com/site/node/47037
     - also see https://bugzilla.redhat.com/show_bug.cgi?id=744052#c4
       (also relevant to Pacemaker)
     - exclusive resource with value of the respective parameter specifying
       priority in the inverse sence (1 is highest, XXX or 0?)

P: driven by utilization constraint
   - 1. no node-exclusiveness model
     - default (no such purposefully full utilization of node resources
       specified)
   - 2. flat model
     - resources (presumably uniformly, but broken down to utilization
       of comprising primitives) require full utilization of what
       resources (presumably uniformly) each node provides
   - 3. pre-emptive model
     - 1. + prioritization defined by the means of priority per primitive,
     - https://access.redhat.com/site/solutions/65542
   - for 2., 3.:
     - however modelling "exclusive resource cannot be started on node,
       with non-exclusive resources already running" seems to be close
       to impossible (XXX or in a intrusive way, like setting default
       utilization + priority for those resources not overriding
       these defaults)?


Other resource properties
=========================

Is-managed resource property
----------------------------

MANAGED ::= MANAGED(RESOURCE, TRUE)
          | MANAGED(RESOURCE, FALSE)

. MANAGED(RESOURCE, TRUE)  ... resource is managed (default assumption)
. MANAGED(RESOURCE, FALSE) ... resource is not managed

R: XXX driven by clusvcadm -Z/-U (or at least it is close)

P: via is-managed meta-attribute


References
==========

[1] http://en.wikipedia.org/wiki/Linear_temporal_logic
: vim: set ft=rst:  <-- not exactly, but better than nothing