path: root/__root__/doc/rgmanager-pacemaker.02.resources.txt

IN THE LIGHT OF RGMANAGER-PACEMAKER CONVERSION: 02/CLUSTERED RESOURCE PROPERTIES

Copyright 2016 Red Hat, Inc., Jan Pokorný <jpokorny @at@ Red Hat .dot. com>
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".


Preface
=======

This document elaborates on how selected resource relationship properties
(denoting the run-time behavior) formalized by the means of LTL logic maps
to particular RGManager (R) and Pacemaker (P) configuration arrangements.
Due to the purpose of this document, "selected" here means set of
properties one commonly uses in case of the former cluster resource
manager (R).

Properties are categorised, each is further dissected based on
the property variants (basically holds or doesn't, but can be more
convoluted), and for each variants, the LTL model and R+P specifics
are provided.


Outline
-------

Resource-resource interaction properites, PROPERTY(RESOURCE1, RESOURCE2)
. ORDERING
. COOCCURRENCE
Relative resource-node assignment properties, PROPERTY(RESOURCE)
. STICKY
. EXCLUSIVE
Explicit resource-node assignment properties, PROPERTY(RESOURCE, NODE)
. AFFINITY
Other resource properties, PROPERTY(RESOURCE)
. RECOVERY  --> see RECOVERY(GROUP) and FAILURE-ISOLATION(GROUP, RESOURCE)
. ENABLED   --> Pacemaker only, same as ENABLED(GROUP)

# XXX: service ref=... + single node failover domains vs. clone



Resource-resource interaction properites
========================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE1, RESOURCE2),
implying modification of the behavior of cluster wrt. resource-resource
pair:

PROPERTY(RESOURCE1, RESOURCE2) -> ALTER((RESOURCE1, RESOURCE2))


Ordering (time-based interaction dependence of the resources/their states)
--------------------------------------------------------------------------

ORDERING ::= ORDERING(RESOURCE1, RESOURCE2, NONE)
           | ORDERING(RESOURCE1, RESOURCE2, WEAK)
           | ORDERING(RESOURCE1, RESOURCE2, STRONG)
           | ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)
. ORDERING(RESOURCE1, RESOURCE2, NONE)   ... no time-based dependency
                                             of the states
. ORDERING(RESOURCE1, RESOURCE2, WEAK)   ... start of 2nd preceeded by start
                                             of 1st (stop viceversa/LIFO),
                                             but only when both events happen
                                             at the same time
. ORDERING(RESOURCE1, RESOURCE2, STRONG) ... runtime of 2nd won't exceed
                                             runtime of 1st
. ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)
                                         ... runtime of 2nd won't exceed
                                             runtime of 1st (as STRONG) only
                                             at the start phase (XXX then
                                             the lifetimes are not
                                             correlated???)

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B in RUNNABLE(A1), C in RUNNABLE(A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... intention RUNNING(A1, B)
. d ... intention RUNNING(A2, C)

R: driven by
   XXX implicit and explicit ordering

P: driven by `order` constraint (or `group` arrangement)

ORDERING(B, C, WEAK)  [2. weak ordering]
~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> (X a OR TRUE) AND X X b  [start: B, then C]
a AND b AND ~c AND ~d -> X ~b AND X X ~a          [stop:  C, then B]

R: TBD

P: driven by specifying `kind` as `Optional` (or `score` as `0`)
   # pcs constraint order B then C kind=Optional
   or
   # pcs constraint order B then C score=0

ORDERING(B, C, STRONG)  [3. strong ordering]
~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]
a AND b AND ~c AND ~d -> X ~b AND X X ~a  [see weak ordering]
a AND b AND ~c -> a AND b AND ~c AND ~d   [stronger, follows as per previous]

R: TBD

P: driven by omitting both `kind` and `score` (or specifying `kind` as
   `Mandatory` or `score` as non-zero)
   or (b) using `group` (which implies also `colocation` constraint)
   - (a)
     # pcs constraint order B then C
     # pcs constraint order B then C kind=Mandatory
     or
     # pcs constraint order B then C score=1
     # XXX pcs constraint order set B C
   - (b)
     # XXX pcs resource group add SOMENAME C/RESOURCE2 B/RESOURCE1

ORDERING(B, C, ASYMMETRIC)  [4. asymmetric ordering]
~~~~~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]

R: TBD

P: driven by specifying `symmetrical` as `false` (default is `true`)
   # pcs constraint order B then C symmetrical=false


Cooccurrence (location-based interaction dependence of the resources)
---------------------------------------------------------------------

COOCCURRENCE ::= COOCURRENCE(RESOURCE1, RESOURCE2, NONE)
               | COOCURRENCE(RESOURCE1, RESOURCE2, POSITIVE)
               | COOCURRENCE(RESOURCE1, RESOURCE2, NEGATIVE)
               | COOCURRENCE(RESOURCE1, RESOURCE2, SCORE),
                 SCORE in {..., -1, 0, 1, ...}, 0~NONE, -INF~NEGATIVE,
                                                +INF~POSITIVE
. COOCCURRENCE(RESOURCE1, RESOURCE2, NONE)     ... not any occurrence
                                                   relationship (default)
. COOCCURRENCE(RESOURCE1, RESOURCE2, POSITIVE) ... positive occurrence
                                                   relationship (flat model)
. COOCCURRENCE(RESOURCE1, RESOURCE2, NEGATIVE) ... negative occurrence
                                                   relationship (flat model)
. COOCCURRENCE(RESOURCE1, RESOURCE2, SCORE)    ... score-based/advisory
                                                   occurrence relationship

note: COOCCURRENCE relation between RESOURCE1 and RESOURCE2 is not symmetric,
      RESOURCE1 is "dependent", RESOURCE2 is "leader"

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B, C in RUNNABLE(A1) intersection RUNNABLE(A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... A1 == A2
. d ... RUNNING(A2, B)

R: driven by various arrangements

P: driven by `colocation` constraint (or `group` arrangement)

COOCCURRENCE(B/RESOURCE1, C/RESOURCE2, NONE)  [1. no cooccurrence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R: default

P: default, no need for that, otherwise specifying `score` as `0`
   # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 0

COOCCURRENCE(B/RESOURCE1, C/RESOURCE2, POSITIVE)  [2. positive cooccurrence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> c  [basic positive cooccurrence condition]
X b -> X d    ["dependent" follows its "leader"]

R: driven by grouping set of sequentially dependent resources
   either using subsequent nesting so as to preserve the predestined
   order unconditionally (referred to as **explicit ordering**)
   or just enumerating them on the same level within the hierarchy
   to allow for (reasonably preselected) reordering as designated
   (via service.sh metadata) rules (referred to as **implicit ordering**)
   hierarchically (in general, but using just the latter provision
   it can be plain flat) into service/vm stanza
   - `__independent_subtree` must not be used

P: driven by (a) `colocation` constraint, specifying `INFINITY`,
   or (b) using `group` (which implies also `order` constraint)
   - (a)
     # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2
     # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 INFINITY
     # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1
   - (b)
     # pcs resource group add SOMENAME C/RESOURCE2 B/RESOURCE1

COOCCURRENCE(B/RESOURCE1, C/RESOURCE2, NEGATIVE)  [3. negative cooccurrence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> ~c  [basic negative cooccurrence condition]
X b -> X ~d    ["dependent" escapes its "leader"]

R: driven/emulated solely by disjunct failover domains
   - XXX and possibly with `follow_service.sl` in `central_processing` mode

P: driven by `colocation` constraint, specifying `-INFINITY`
   - using -INFINITY as value
   # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 -INFINITY
   # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1 setoptions score=-INFINITY

COOCURENCE(B/RESOURCE1, C/RESOURCE2, SCORE)  [4. score-based/advisory occurrence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TBD

R: XXX not supported unless a way to emulate this using `central_processing`
  (?) or sharing the same `ordered` failover domain (?)

P: driven by `colocation` constraint, specifying `SCORE` (value)
   # pcs constraint colocation add B/RESOURCE1 with c/RESOURCE2 SCORE
   # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1 setoptions score=SCORE



Relative resource-node assignment properties
============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE),
implying modification of the behavior of cluster wrt. the node
running RESOURCE:

PROPERTY(RESOURCE) AND RUNNING(NODE, RESOURCE) -> ALTER(NODE)


Resource stickiness property (not moving back to preferred location)
--------------------------------------------------------------------

STICKY ::= STICKY(RESOURCE, FALSE)
         | STICKY(RESOURCE, TRUE)
         | STICKY(RESOURCE, STICKINESS), STICKINESS in {0, 1, 2, ...},
                                         0~FALSE
. STICKY(RESOURCE, FALSE)      ... unsticky/cruising resource (default)
. STICKY(RESOURCE, TRUE)       ... sticky resource (flat model)
. STICKY(RESOURCE, STICKINESS) ... sticky resource (prioritized model)

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B in RUNNABLE(A1) AND B in RUNNABLE(A2)
. SCORE(B, A1) < SCORE(B, A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... ACTIVE(A2)
. c ... RUNNING(A2, B) resource X running (relocation if A1 != A2)

R: driven by `/cluster/rm/failoverdomains/failoverdomain/@nofailback`
   - note: only applies to service/vm (not primitive resources)
P: driven by `stickiness` parameter
   - group is a sum of stickiness values of underlying resources


STICKY(B/RESOURCE, FALSE)  [1. model of unsticky/cruising resource]
~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X c

R: default, no need for that, otherwise specifying `@nofailback` as `0`

P: default, no need for that, otherwise specifying `stickiness` as `0`
   # pcs resource meta B/RESOURCE stickiness=
   # pcs resource meta B/RESOURCE stickiness=0

STICKY(B/RESOURCE, TRUE)  [2. model of sticky resource]
~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X a

R: driven by specifying `@nofailback` as positive number

P: driven by specifying `stickiness` as `INFINITY`
   # pcs resource meta B/RESOURCE stickiness=INFINITY

STICKY(B/RESOURCE, STICKINESS)  [3. model of sticky resource with priorities]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TBD

R: XXX not supported unless a way to emulate this using ordered failover
   domains(?)

P: driven by specifying `stickiness` as `STICKINESS` (value)
   # pcs resource meta B/RESOURCE stickiness=STICKINESS


Node-exclusiveness resource property (optionally with priority-based preemption)
--------------------------------------------------------------------------------

EXCLUSIVE ::= EXCLUSIVE(RESOURCE, FALSE)
            | EXCLUSIVE(RESOURCE, TRUE)
            | EXCLUSIVE(RESOURCE, PRIORITY), PRIORITY in {0, 1, ...}, 0~FALSE
. EXCLUSIVE(RESOURCE, FALSE)    ... non-exclusive resource (default)
. EXCLUSIVE(RESOURCE, TRUE)     ... exclusive (flat model)
. EXCLUSIVE(RESOURCE, PRIORITY) ... exclusive (prioritized pre-emptive model)

assumed variables (+ constraints) to be combined with proper preconditions:
. A in NODES
. B, C in RUNNABLE(A)
. I, J in {0, 1, ...}: I > J

R: driven by `/cluster/rm/<service>/@exclusive`
   - note: only applies to service/vm (not primitive resources)
P: driven by `utilization` constraint
   (or possibly by a set of `colocation` constraints:
   for all r in RESOURCES\RESOURCE: COOCCURRENCE(RESOURCE, r, -INF) [1]
   (see [3. negative cooccurrence])

[1. model of non-node-exclusive/co-occurrence-positive resource, based on 2.]
for all B' in RUNNABLE(A): EXCLUSIVE(B', FALSE) [implies EXCLUSIVE(B, FALSE)]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~(a OR b) OR ~(~a -> ~b R ~a AND ~b -> ~a R ~b)
= ~(a OR b) OR ~(~a -> ~b R ~a) OR ~(~b -> ~a R ~b)
= ~(a OR b) OR ~(a OR ~b R ~a) OR ~(b OR ~a R ~b)
= ~(a OR b) OR ~a AND ~(~b R ~a) OR ~b AND ~(~a R ~b)
= ~(a OR b) OR ~a AND ~(~(a U b)) OR ~b AND ~(~(b U a))
= ~(a OR b) OR ~a AND a U b OR ~b AND b U a
= ~(a OR b) OR b OR a
= ~a AND ~b OR b OR a [= true, i.e. no restriction wrt. modelled property, QED]
. a ... RUNNING(A, B)
. b ... exists B' in 2^RUNNABLE(A)\{}: RUNNING(A, B')

R: default (`@exclusive` = 0)

P: default (no such purposefully full utilization of node resources
            specified)
   - pcs: support is arriving
     (https://bugzilla.redhat.com/show_bug.cgi?id=1158500)

[2. flat model of node-exclusive/co-occurrence-less resource, no preemption]
EXCLUSIVE(B, TRUE)
~~~~~~~~~~~~~~~~~~
~(a OR b) OR (~a -> ~b R ~a) AND (~b -> ~a R ~b)  [mutual exclusion]
. a ... RUNNING(A, B)
. b ... exists YS in 2^(RUNNABLE(A)\{B})\{}: for all Y in YS: RUNNING(A, Y)
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

R: driven by the arrangement:
   - not `central_processing mode` and `@exclusive` != 0

P: driven by the arrangement:
   - resources (presumably uniformly, but broken down to utilization
     of comprising primitives) require full utilization of what
     resources (presumably uniformly) each node provides

[3. model of prioritized pre-emptive node-exclusive resource]
EXCLUSIVE(B, I)
EXCLUSIVE(C, J)
~~~~~~~~~~~~~~~
~(a OR c) OR (~a -> ~c R ~a) AND (~c -> ~a R ~c),  [mutual exclusion]
(b AND X a) -> (X ~b)                              [exlusivness priority wins]
. a ... RUNNING(A, B)
. b ... RUNNING(C, B)
. c ... exists YS in 2^({Y | Y in RUNNABLE(A)
                             AND (EXCLUSIVE(Y, FALSE)
                                  OR exists K > I: EXCLUSIVE(Y, K))}\{B}
                        )\{}: for all Y' in YS: RUNNING(A, Y')
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

R: driven by the arrangement:
   - only in `central_processing mode`
   - https://access.redhat.com/site/node/47037
   - also see https://bugzilla.redhat.com/show_bug.cgi?id=744052#c4
     (also relevant to Pacemaker)
   - exclusive resource with value of the respective parameter specifying
     priority in the inverse sense (1 is highest, XXX or 0?)

P: driven by the arrangement:
   - see 1. + prioritization defined by the means of priority per primitive,
     - https://access.redhat.com/site/solutions/65542
   - for 2., 3.:
     - however modelling "exclusive resource cannot be started on node,
       with non-exclusive resources already running" seems to be close
       to impossible (XXX or in a intrusive way, like setting default
       utilization + priority for those resources not overriding
       these defaults)?



Explicit resource-node(s) assignment property
=============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE, NODE),
implying modification of the behavior of cluster wrt. resource-node
pair:

PROPERTY(RESOURCE, NODE) -> ALTER((RESOURCE, NODE))


Resource-node affinity
----------------------

AFFINITY ::= AFFINITY(RESOURCE, NODE, NONE)
           | AFFINITY(RESOURCE, NODE, FALSE)
           | AFFINITY(RESOURCE, NODE, TRUE)
           | AFFINITY(RESOURCE, NODE, WEIGHT), WEIGHT in {..., -1, 0, 1, ...},
                                               0~NONE, -INF~FALSE, +INF~TRUE
. AFFINITY(RESOURCE, NODE, NONE)   ... no special affinity (default)
. AFFINITY(RESOURCE, NODE, FALSE)  ... anti-affinity (the node cannot run)
. AFFINITY(RESOURCE, NODE, TRUE)   ... node forms a set of executive nodes
. AFFINITY(RESOURCE, NODE, WEIGHT) ... prioritized model

assumed variables (+ constraints) to be combined with proper preconditions:
. AN in NODES^2\{}
. B in intersection( { RUNNABLE(A) | A in AN } )

propositional variables:
. a ... RUNNING(A', B)
. b ... A' in AN
. c ... STICKY(B, FALSE)  <-- see above
. d ... A' in {A'' | A'' in AN: WEIGHT(B, A'') == max({WEIGHT(B, A''')
                                                       | A''' in AN})}

R: driven by the failoved domains arrangement for given service/vm
   - XXX check that RGManager indeed behaves like Pacemaker's
     `symmetric-cluster`
P: driven by `location` constraint
   - only default `symmetric-cluster` is considered

exists A in AN: AFFINITY(B, A, NONE)  [1. no resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R: driven by general arrangement patterns (possibilities with "NONE"):
   - if for all A in AN: AFFINITY(B, A, NONE):
     - nodes AN form attached failover domain, and this is either not `ordered`
       or with the same `priority` for ordering
     - alternatively, service/vm B has no failoverdomain explicitly attached
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, TRUE):
     - nodes AN for which A in AN: AFFINITY(B, A, TRUE) form attached failover
       domain, and this is not `restricted` (to allow those "NONE" nodes to
       be ever used)
     - or nodes AN form attached failover domain, and it is `ordered` with
       two levels of priorities (for TRUE higher, for NONE lower)
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, FALSE):
     - nodes AN for which A in AN: AFFINITY(B, A, NONE) form attached failover
       domain, and this is either not `ordered` or with the same priority for
       ordering, and has to be `restricted` to avoid slipping nodes
       A in AN: AFFINITY(B, A, FALSE)
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, FALSE)
                              OR AFFINITY(B, A, TRUE):
     - intersection of the previous two cases
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, WEIGHT),
     -INFINITY < WEIGHT < INFINITY:
     - either any WEIGHT < 0 becomes truncated to 0 (hence dropping losing
       part of the configuration), or if such one present, the whole range
       is rescaled to 0 < WEIGHT < INFINITY/numeric equivalent
       and those member nodes having no affinity in this case have
       lowest priority (hence highest `priority` value)
     - either way, leads to AN forming attached failover domain, and it has
       to be `ordered` with priorities as per previous point
   - else?

P: default, no need for that, otherwise specifying `score` as `0`
   # pcs constraint location B prefers A= (??? to remove it as such)
   # pcs constraint location B prefers A=0
   # pcs contraint location add SOMEID B A 0

exists A in AN: AFFINITY(B, A, FALSE)  [2. antagonist resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X ~b

R: driven by this arrangement:
   - if for all A in AN: AFFINITY(B, A, FALSE):
     - nodes NODES\AN form attached failover domain, which has to
       be `restricted` to prevent slipping onto any node in AN
   - else:
     - combine using patterns as in 1.

P: by specifying (directly or indirectly) `score` as `-INFINITY`
   # pcs constraint location B avoids A
   # pcs constraint location B avoids A=INFINITY
   # pcs constraint location B prefers A=-INFINITY (???)
   # pcs contraint location add SOMEID B A -INFINITY
   # crm_resource --ban --resource B --host A ...

exists A in AN: AFFINITY(B, A, TRUE)  [3. (positive) resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X b

R: driven by this arrangement:
   - if for all A in AN: AFFINITY(B, A, TRUE):
     - nodes AN form selected failover domain
   - else:
     - combine using patterns as in 1.

P: by specifying (directly or indirectly) `score` as `INFINITY`
   # pcs constraint location B prefers A
   # pcs constraint location B prefers A=INFINITY
   # pcs constraint location B avoids A=-INFINITY (???)
   # pcs contraint location add SOMEID B A INFINITY

[4. model of resource-node affinity with priorities]
for all A in AN: AFFINITY(B, A, -INFINITY < WEIGHT(B,A) < INFINITY)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a AND c -> X d  [when not sticky, resource will run on the node
                   this resource has greatest affinity to]

R: driven by this arrangement:
   - see the respective pattern at 1.

P: by specifying `score` as WEIGHT
   # pcs constraint location B {prefers|avoids} A=WEIGHT(B,A)
   # pcs constraint location add SOMEID B A WEIGHT(B,A)


Initial exclusive resource-node affinity
----------------------------------------

TBD:
https://bugzilla.redhat.com/994215#RFE--Control-which-node-a-service-autostarts-on



References
==========

[1] http://oss.clusterlabs.org/pipermail/users/2016-January/002197.html

: vim: set ft=rst:  <-- not exactly, but better than nothing