path: root/__root__/doc/rgmanager-pacemaker.02.resources.txt

IN THE LIGHT OF RGMANAGER-PACEMAKER CONVERSION: 02/CLUSTERED RESOURCE PROPERTIES

Copyright 2016 Red Hat, Inc., Jan Pokorný <jpokorny @at@ Red Hat .dot. com>
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".


Preface
=======

This document elaborates on how selected resource relationship properties
(denoting the run-time behavior) formalized by the means of LTL logic maps
to particular RGManager (R) and Pacemaker (P) configuration arrangements.
Due to the purpose of this document, "selected" here means set of
properties one commonly uses in case of the former cluster resource
manager (R).

Properties are categorised, each is further dissected based on
the property variants (basically holds or doesn't, but can be more
convoluted), and for each variants, the LTL model and R+P specifics
are provided.


Outline
-------

Resource-resource interaction properites, PROPERTY(RESOURCE1, RESOURCE2)
. ORDERING
. COOCCURENCE
Relative resource-node assignment properties, PROPERTY(RESOURCE)
. STICKY
. EXCLUSIVE
Explicit resource-node assignment properties, PROPERTY(RESOURCE, NODE)
. AFFINITY
Other resource properties, PROPERTY(RESOURCE)
. RECOVERY
. MANAGED

# XXX: service ref=... + single node failover domains vs. clone



Resource-resource interaction properites
========================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE1, RESOURCE2),
implying modification of the behavior of cluster wrt. resource-resource
pair:

PROPERTY(RESOURCE1, RESOURCE2) -> ALTER((RESOURCE1, RESOURCE2))


Ordering (time-based interaction dependence of the resources/their states)
--------------------------------------------------------------------------

ORDERING ::= ORDERING(RESOURCE1, RESOURCE2, NONE)
           | ORDERING(RESOURCE1, RESOURCE2, WEAK)
           | ORDERING(RESOURCE1, RESOURCE2, STRONG)
           | ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)
. ORDERING(RESOURCE1, RESOURCE2, NONE)   ... no time-based dependency
                                             of the states
. ORDERING(RESOURCE1, RESOURCE2, WEAK)   ... start of 2nd preceeded by start
                                             of 1st (stop viceversa/LIFO),
                                             but only when both events happen
                                             at the same time
. ORDERING(RESOURCE1, RESOURCE2, STRONG) ... runtime of 2nd won't exceed
                                             runtime of 1st
. ORDERING(RESOURCE1, RESOURCE2, ASYMMETRIC)
                                         ... runtime of 2nd won't exceed
                                             runtime of 1st (as STRONG) only
                                             at the start phase (XXX then
                                             the lifetimes are not
                                             correlated???)

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B in RUNNABLE(A1), C in RUNNABLE(A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... intention RUNNING(A1, B)
. d ... intention RUNNING(A2, C)

R: driven by
   XXX implicit and explicit ordering

P: driven by `order` constraint (or `group` arrangement)

ORDERING(B, C, WEAK)  [2. weak ordering]
~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> (X a OR TRUE) AND X X b  [start: B, then C]
a AND b AND ~c AND ~d -> X ~b AND X X ~a          [stop:  C, then B]

R: TBD

P: driven by specifying `score` as `0`
   # pcs constraint order B then C score=0

ORDERING(B, C, STRONG)  [3. strong ordering]
~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]
a AND b AND ~c AND ~d -> X ~b AND X X ~a  [see weak ordering]
a AND b AND ~c -> a AND b AND ~c AND ~d   [stronger, follows as per previous]

R: TBD

P: driven by omitting `score` (or specifying it as non-zero)
   or (b) using `group` (which implies also `colocation` constraint)
   - (a)
     # pcs constraint order B then C
     # pcs constraint order B then C score=1
     # XXX pcs constraint order set B C
   - (b)
     # XXX pcs resource group add SOMENAME C/RESOURCE2 B/RESOURCE1

ORDERING(B, C, ASYMMETRIC)  [4. asymmetric ordering]
~~~~~~~~~~~~~~~~~~~~~~~~~~
~a AND ~b AND c AND d -> X a AND X X b    [see weak ordering]
~a AND ~b AND d -> ~a AND ~b AND c AND d  [stronger, follows as per previous???]

R: TBD

P: driven by specifying `symmetrical` as `false` (default is `true`)
   # pcs constraint order B then C symmetrical=false


Cooccurence (location-based interaction dependence of the resources)
--------------------------------------------------------------------

COOCCURENCE ::= COOCURENCE(RESOURCE1, RESOURCE2, NONE)
              | COOCURENCE(RESOURCE1, RESOURCE2, POSITIVE)
              | COOCURENCE(RESOURCE1, RESOURCE2, NEGATIVE)
              | COOCURENCE(RESOURCE1, RESOURCE2, SCORE),
                SCORE in {..., -1, 0, 1, ...}, 0~NONE, -INF~NEGATIVE,
                                               +INF~POSITIVE
. COOCCURENCE(RESOURCE1, RESOURCE2, NONE)     ... not any occurence
                                                  relationship (default)
. COOCCURENCE(RESOURCE1, RESOURCE2, POSITIVE) ... positive occurence
                                                  relationship (flat model)
. COOCCURENCE(RESOURCE1, RESOURCE2, NEGATIVE) ... negative occurence
                                                  relationship (flat model)
. COOCCURENCE(RESOURCE1, RESOURCE2, SCORE)    ... score-based/advisory
                                                  occurence relationship

note: COOCCURENCE relation between RESOURCE1 and RESOURCE2 is not symmetric,
      RESOURCE1 is "dependent", RESOURCE2 is "leader"

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B, C in RUNNABLE(A1) intersection RUNNABLE(A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... RUNNING(A2, C)
. c ... A1 == A2
. d ... RUNNING(A2, B)

R: driven by various arrangements

P: driven by `colocation` constraint (or `group` arrangement)

COOCCURENCE(B/RESOURCE1, C/RESOURCE2, NONE)  [1. no cooccurence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R: default

P: default, no need for that, otherwise specifying `score` as `0`
   # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 0

COOCCURENCE(B/RESOURCE1, C/RESOURCE2, POSITIVE)  [2. possitive cooccurence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> c  [basic positive cooccurence condition]
X b -> X d    ["dependent" follows its "leader"]

R: driven by grouping set of sequentially dependent resources
   either using subsequent nesting so as to preserve the predestined
   order unconditionally (referred to as **explicit ordering**)
   or just enumerating them on the same level within the hierarchy
   to allow for (reasonably preselected) reordering as designated
   (via service.sh metadata) rules (referred to as **implicit ordering**)
   hierarchically (in general, but using just the latter provision
   it can be plain flat) into service/vm stanza
   - `__independent_subtree` must not be used

P: driven by (a) `colocation` constraint, specifying `INFINITY`,
   or (b) using `group` (which implies also `order` constraint)
   - (a)
     # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2
     # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 INFINITY
     # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1
   - (b)
     # pcs resource group add SOMENAME C/RESOURCE2 B/RESOURCE1

COOCCURENCE(B/RESOURCE1, C/RESOURCE2, NEGATIVE)  [3. negative cooccurence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> ~c  [basic negative cooccurence condition]
X b -> X ~d    ["dependent" escapes its "leader"]

R: driven/emulated solely by disjunct failover domains
   - XXX and possibly with `follow_service.sl` in `central_processing` mode

P: driven by `colocation` constraint, specifying `-INFINITY`
   - using -INFINITY as value
   # pcs constraint colocation add B/RESOURCE1 with C/RESOURCE2 -INFINITY
   # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1 setoptions score=-INFINITY

COOCURENCE(B/RESOURCE1, C/RESOURCE2, SCORE)  [4. score-based/advisory occurence]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TBD

R: XXX not supported unless a way to emulate this using `central_processing`
  (?) or sharing the same `ordered` failover domain (?)

P: driven by `colocation` constraint, specifying `SCORE` (value)
   # pcs constraint colocation add B/RESOURCE1 with c/RESOURCE2 SCORE
   # pcs constraint colocation set C/RESOURCE2 B/RESOURCE1 setoptions score=SCORE



Relative resource-node assignment properties
============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE),
implying modification of the behavior of cluster wrt. the node
running RESOURCE:

PROPERTY(RESOURCE) AND RUNNING(NODE, RESOURCE) -> ALTER(NODE)


Resource stickiness property (not moving back to preferred location)
--------------------------------------------------------------------

STICKY ::= STICKY(RESOURCE, FALSE)
         | STICKY(RESOURCE, TRUE)
         | STICKY(RESOURCE, STICKINESS), STICKINESS in {0, 1, 2, ...},
                                         0~FALSE
. STICKY(RESOURCE, FALSE)      ... unsticky/cruising resource (default)
. STICKY(RESOURCE, TRUE)       ... sticky resource (flat model)
. STICKY(RESOURCE, STICKINESS) ... sticky resource (prioritized model)

assumed variables (+ constraints) to be combined with proper preconditions:
. A1, A2 in NODES
. B in RUNNABLE(A1) AND B in RUNNABLE(A2)
. SCORE(B, A1) < SCORE(B, A2)

propositional variables:
. a ... RUNNING(A1, B)
. b ... ACTIVE(A2)
. c ... RUNNING(A2, B) resource X running (relocation if A1 != A2)

R: driven by `/cluster/rm/failoverdomains/failoverdomain/@nofailback`
   - note: only applies to service/vm (not primitive resources)
P: driven by `stickiness` parameter
   - group is a sum of stickiness values of underlying resources


STICKY(B/RESOURCE, FALSE)  [1. model of unsticky/cruising resource]
~~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X c

R: default, no need for that, otherwise specifying `@nofailback` as `0`

P: default, no need for that, otherwise specifying `stickiness` as `0`
   # pcs resource meta B/RESOURCE stickiness=
   # pcs resource meta B/RESOURCE stickiness=0

STICKY(B/RESOURCE, TRUE)  [2. model of sticky resource]
~~~~~~~~~~~~~~~~~~~~~~~~
a AND b -> X a

R: driven by specifying `@nofailback` as positive number

P: driven by specifying `stickiness` as `INFINITY`
   # pcs resource meta B/RESOURCE stickiness=INFINITY

STICKY(B/RESOURCE, STICKINESS)  [3. model of sticky resource with priorities]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TBD

R: XXX not supported unless a way to emulate this using ordered failover
   domains(?)

P: driven by specifying `stickiness` as `STICKINESS` (value)
   # pcs resource meta B/RESOURCE stickiness=STICKINESS


Node-exclusiveness resource property (optionally with priority-based preemption)
--------------------------------------------------------------------------------

EXCLUSIVE ::= EXCLUSIVE(RESOURCE, FALSE)
            | EXCLUSIVE(RESOURCE, TRUE)
            | EXCLUSIVE(RESOURCE, PRIORITY), PRIORITY in {0, 1, ...}, 0~FALSE
. EXCLUSIVE(RESOURCE, FALSE)    ... non-exclusive resource (default)
. EXCLUSIVE(RESOURCE, TRUE)     ... exclusive (flat model)
. EXCLUSIVE(RESOURCE, PRIORITY) ... exclusive (prioritized pre-emptive model)

assumed variables (+ constraints) to be combined with proper preconditions:
. A in NODES
. B, C in RUNNABLE(A)
. I, J in {0, 1, ...}: I > J

R: driven by `/cluster/rm/<service>/@exclusive`
   - note: only applies to service/vm (not primitive resources)
P: driven by `utilization` constraint
   (or possibly by a set of `colocation` constraints:
   for all r in RESOURCES\RESOURCE: COOCCURENCE(RESOURCE, r, -INF) [1]

[1. model of non-node-exclusive/co-occurence-positive resource, based on 2.]
for all B' in RUNNABLE(A): EXCLUSIVE(B', FALSE) [implies EXCLUSIVE(B, FALSE)]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~(a OR b) OR ~(~a -> ~b R ~a AND ~b -> ~a R ~b)
= ~(a OR b) OR ~(~a -> ~b R ~a) OR ~(~b -> ~a R ~b)
= ~(a OR b) OR ~(a OR ~b R ~a) OR ~(b OR ~a R ~b)
= ~(a OR b) OR ~a AND ~(~b R ~a) OR ~b AND ~(~a R ~b)
= ~(a OR b) OR ~a AND ~(~(a U b)) OR ~b AND ~(~(b U a))
= ~(a OR b) OR ~a AND a U b OR ~b AND b U a
= ~(a OR b) OR b OR a
= ~a AND ~b OR b OR a [= true, i.e. no restriction wrt. modelled property, QED]
. a ... RUNNING(A, B)
. b ... exists B' in 2^RUNNABLE(A)\{}: RUNNING(A, B')

R: default (`@exclusive` = 0)

P: default (no such purposefully full utilization of node resources
            specified)
   - pcs: support is arriving
     (https://bugzilla.redhat.com/show_bug.cgi?id=1158500)

[2. flat model of node-exclusive/co-occurence-less resource, no preemption]
EXCLUSIVE(B, TRUE)
~~~~~~~~~~~~~~~~~~
~(a OR b) OR (~a -> ~b R ~a) AND (~b -> ~a R ~b)  [mutual exclusion]
. a ... RUNNING(A, B)
. b ... exists YS in 2^(RUNNABLE(A)\{B})\{}: for all Y in YS: RUNNING(A, Y)
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

R: driven by the arrangement:
   - not `central_processing mode` and `@exclusive` != 0

P: driven by the arrangement:
   - resources (presumably uniformly, but broken down to utilization
     of comprising primitives) require full utilization of what
     resources (presumably uniformly) each node provides

[3. model of prioritized pre-emptive node-exclusive resource]
EXCLUSIVE(B, I)
EXCLUSIVE(C, J)
~~~~~~~~~~~~~~~
~(a OR c) OR (~a -> ~c R ~a) AND (~c -> ~a R ~c),  [mutual exclusion]
(b AND X a) -> (X ~b)                              [exlusivness priority wins]
. a ... RUNNING(A, B)
. b ... RUNNING(C, B)
. c ... exists YS in 2^({Y | Y in RUNNABLE(A)
                             AND (EXCLUSIVE(Y, FALSE)
                                  OR exists K > I: EXCLUSIVE(Y, K))}\{B}
                        )\{}: for all Y' in YS: RUNNING(A, Y')
        (assuming valid state, i.e., exclusiveness property
         would be recursively satisfied also within YS)

R: driven by the arrangement:
   - only in `central_processing mode`
   - https://access.redhat.com/site/node/47037
   - also see https://bugzilla.redhat.com/show_bug.cgi?id=744052#c4
     (also relevant to Pacemaker)
   - exclusive resource with value of the respective parameter specifying
     priority in the inverse sense (1 is highest, XXX or 0?)

P: driven by the arrangement:
   - see 1. + prioritization defined by the means of priority per primitive,
     - https://access.redhat.com/site/solutions/65542
   - for 2., 3.:
     - however modelling "exclusive resource cannot be started on node,
       with non-exclusive resources already running" seems to be close
       to impossible (XXX or in a intrusive way, like setting default
       utilization + priority for those resources not overriding
       these defaults)?



Explicit resource-node(s) assignment property
=============================================

Generally a relation expressed by a predicate PROPERTY(RESOURCE, NODE),
implying modification of the behavior of cluster wrt. resource-node
pair:

PROPERTY(RESOURCE, NODE) -> ALTER((RESOURCE, NODE))


Resource-node affinity
----------------------

AFFINITY ::= AFFINITY(RESOURCE, NODE, NONE)
           | AFFINITY(RESOURCE, NODE, FALSE)
           | AFFINITY(RESOURCE, NODE, TRUE)
           | AFFINITY(RESOURCE, NODE, WEIGHT), WEIGHT in {..., -1, 0, 1, ...},
                                               0~NONE, -INF~FALSE, +INF~TRUE
. AFFINITY(RESOURCE, NODE, NONE)   ... no special affinity (default)
. AFFINITY(RESOURCE, NODE, FALSE)  ... anti-affinity (the node cannot run)
. AFFINITY(RESOURCE, NODE, TRUE)   ... node forms a set of executive nodes
. AFFINITY(RESOURCE, NODE, WEIGHT) ... prioritized model

assumed variables (+ constraints) to be combined with proper preconditions:
. AN in NODES^2\{}
. B in intersection( { RUNNABLE(A) | A in AN } )

propositional variables:
. a ... RUNNING(A', B)
. b ... A' in AN
. c ... STICKY(B, FALSE)  <-- see above
. d ... A' in {A'' | A'' in AN: WEIGHT(B, A'') == max({WEIGHT(B, A''')
                                                       | A''' in AN})}

R: driven by the failoved domains arrangement for given service/vm
   - XXX check that RGManager indeed behaves like Pacemaker's
     `symmetric-cluster`
P: driven by `location` constraint
   - only default `symmetric-cluster` is considered

exists A in AN: AFFINITY(B, A, NONE)  [1. no resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R: driven by general arrangement patterns (possibilities with "NONE"):
   - if for all A in AN: AFFINITY(B, A, NONE):
     - nodes AN form attached failover domain, and this is either not `ordered`
       or with the same `priority` for ordering
     - alternatively, service/vm B has no failoverdomain explicitly attached
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, TRUE):
     - nodes AN for which A in AN: AFFINITY(B, A, TRUE) form attached failover
       domain, and this is not `restricted` (to allow those "NONE" nodes to
       be ever used)
     - or nodes AN form attached failover domain, and it is `ordered` with
       two levels of priorities (for TRUE higher, for NONE lower)
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, FALSE):
     - nodes AN for which A in AN: AFFINITY(B, A, NONE) form attached failover
       domain, and this is either not `ordered` or with the same priority for
       ordering, and has to be `restricted` to avoid slipping nodes
       A in AN: AFFINITY(B, A, FALSE)
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, FALSE)
                              OR AFFINITY(B, A, TRUE):
     - intersection of the previous two cases
   - else if for all A in AN: AFFINITY(B, A, NONE) OR AFFINITY(B, A, WEIGHT),
     -INFINITY < WEIGHT < INFINITY:
     - either any WEIGHT < 0 becomes truncated to 0 (hence dropping losing
       part of the configuration), or if such one present, the whole range
       is rescaled to 0 < WEIGHT < INFINITY/numeric equivalent
       and those member nodes having no affinity in this case have
       lowest priority (hence highest `priority` value)
     - either way, leads to AN forming attached failover domain, and it has
       to be `ordered` with priorities as per previous point
   - else?

P: default, no need for that, otherwise specifying `score` as `0`
   # pcs constraint location B prefers A= (??? to remove it as such)
   # pcs constraint location B prefers A=0
   # pcs contraint location add SOMEID B A 0

exists A in AN: AFFINITY(B, A, FALSE)  [2. antagonist resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X ~b

R: driven by this arrangement:
   - if for all A in AN: AFFINITY(B, A, FALSE):
     - nodes NODES\AN form attached failover domain, which has to
       be `restricted` to prevent slipping onto any node in AN
   - else:
     - combine using patterns as in 1.

P: by specifying (directly or indirectly) `score` as `-INFINITY`
   # pcs constraint location B avoids A
   # pcs constraint location B avoids A=INFINITY
   # pcs constraint location B prefers A=-INFINITY (???)
   # pcs contraint location add SOMEID B A -INFINITY
   # crm_resource --ban --resource B --host A ...

exists A in AN: AFFINITY(B, A, TRUE)  [3. (positive) resource-node affinity]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a -> X b

R: driven by this arrangement:
   - if for all A in AN: AFFINITY(B, A, TRUE):
     - nodes AN form selected failover domain
   - else:
     - combine using patterns as in 1.

P: by specifying (directly or indirectly) `score` as `INFINITY`
   # pcs constraint location B prefers A
   # pcs constraint location B prefers A=INFINITY
   # pcs constraint location B avoids A=-INFINITY (???)
   # pcs contraint location add SOMEID B A INFINITY

[4. model of resource-node affinity with priorities]
for all A in AN: AFFINITY(B, A, -INFINITY < WEIGHT(B,A) < INFINITY)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X a AND c -> X d  [when not sticky, resource will run on the node
                   this resource has greatest affinity to]

R: driven by this arrangement:
   - see the respective pattern at 1.

P: by specifying `score` as WEIGHT
   # pcs constraint location B {prefers|avoids} A=WEIGHT(B,A)
   # pcs constraint location add SOMEID B A WEIGHT(B,A)


Initial exclusive resource-node affinity
----------------------------------------

TBD:
https://bugzilla.redhat.com/994215#RFE--Control-which-node-a-service-autostarts-on



Other resource properties
=========================

Recovery policy resource property
---------------------------------

RECOVERY ::= RECOVERY(RESOURCE, RESTART-ONLY)
           | RECOVERY(RESOURCE, RESTART-UNTIL1, MAX-RESTARTS)
           | RECOVERY(RESOURCE, RESTART-UNTIL2, MAX-RESTARTS, EXPIRE-TIME)
           | RECOVERY(RESOURCE, RELOCATE)
           | RECOVERY(RESOURCE, DISABLE)
. RECOVERY(RESOURCE, RESTART)  ... "attempt to restart in place", unlimited
. RECOVERY(RESOURCE, RESTART-UNTIL1, MAX-RESTARTS)
                               ... ditto, but after MAX-RESTARTS attempts
                                   (for the whole period of resource-node
                                   assignment) attempt to relocate
. RECOVERY(RESOURCE, RESTART-UNTIL2, MAX-RESTARTS, EXPIRE-TIME)
                               ... ditto, but after MAX-RESTARTS attempts
                                   accumulated within EXPIRE-TIME windows,
                                   attempt to relocate
. RECOVERY(RESOURCE, RELOCATE) ... move to another node
. RECOVERY(RESOURCE, DISABLE)  ... do not attempt anything, stop

R: driven by `/cluster/rm/(service|vm)/@recovery`

P: driven by OCF RA return code and/or `migration-threshold`

RECOVERY(RESOURCE, RESTART-ONLY)  [1. restart in place, unlimited]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

R: default, no need for that, otherwise specifying `@recovery` as `restart`
   (and not specifying none of `@max_restarts`, `@restart_expire_time`,
   or keeping `@max_restarts` at zero!)

P: default, no need for that, otherwise specifying `migration-threshold`
   as `INFINITY` (or zero?; can be overriden by OCF RA return code, anyway?)

RECOVERY(RESOURCE, RESTART-UNTIL1, MAX-RESTARTS)  [2. restart + absolute limit]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

R: driven by specifying `@max_restarts` as `MAX-RESTARTS` (value, non-positive
   number boils down to case 1.)
   - and, optionally, specifying `@recovery` as `restart` (or not at all!)

P: driven by specifying `migration-threshold` as `MAX-RESTARTS` (value,
   presumably non-negative, `INFINITY` or zero? boil down to case 1.)
   (but can be overriden by OCF RA return code, anyway?)

[3. restart + relative limit for number of restarts/period]
RECOVERY(RESOURCE, RESTART-UNTIL2, MAX-RESTARTS, EXPIRE-TIME)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

R: driven by specifying `@max_restarts` as `MAX-RESTARTS` (value, non-positive
   number boils down to case 1.) and `@restart_expire_time`
   as `EXPIRE-TIME` (value, negative after expansion boils down to the
   case 1., zero to case 2.)
   - and, optionally, specifying `@recovery` as `restart` (or not at all!)

P: driven by specifying `migration-threshold` as `MAX-RESTARTS` (value,
   presumably non-negative, `INFINITY` or zero? boil down to case 1.) and
   `failure-timeout` as `EXPIRE-TIME`  (value, presumably positive, zero
   boils down to case 2.)
   (but can be overriden by OCF RA return code, anyway?)

RECOVERY(RESOURCE, RELOCATE)  [4. move to another node]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

R: driven by specifying `@recovery` as `relocate`

P: driven by specifying `migration-threshold` as 1
   (or possibly negative number?; regardless of `failure-timeout`)
   (but can be overriden by OCF RA return code, anyway?)

RECOVERY(RESOURCE, DISABLE)  [5. no more attempt]
~~~~~~~~~~~~~~~~~~~~~~~~~~~

R: driven by specifying `@recovery` as `disable`

P: can only be achieved in case of AFFINITY(RESOURCE, NODE, FALSE)
   for all nodes except one and specifying `migration-threshold`
   as `1` because upon single failure, remaining
   AFFINITY(RESOURCE, NODE, FALSE) rule for yet-enabled NODE will
   be added, effectively preventing RESOURCE to run anywhere


Is-managed resource property
----------------------------

MANAGED ::= MANAGED(RESOURCE, TRUE)
          | MANAGED(RESOURCE, FALSE)
. MANAGED(RESOURCE, TRUE)   ... resource is managed (default assumption)
. MANAGED(RESOURCE, FALSE)  ... resource is not managed

notes
. see also 01/cluster: FUNCTION

R: except for static un-managing of everything (RGManager avoidance),
   can be partially driven by `/cluster/rm/(service|vm)/@autostart`
   and/or run-time modification using `clusvcadm`
   (or at least it is close???)

P: via `is-managed` meta-attribute

MANAGED(RESOURCE, TRUE)  [1. resource is managed]
~~~~~~~~~~~~~~~~~~~~~~~

R: (partially) driven by specifying `@autostart` as non-zero
   (has to be sequence of digits for sure, though!)
   - default, no need for that
   # clusvcadm -U RESOURCE  <-- whole service/vm only

P: default, no need for that, otherwise specifying `is-managed` as `true`
   # pcs resource manage RESOURCE
   # pcs resource meta RESOURCE is-managed=
   # pcs resource meta RESOURCE is-managed=true

MANAGED(RESOURCE, FALSE)  [2. resource is not managed]
~~~~~~~~~~~~~~~~~~~~~~~~

R: (partially?) driven by specifying `@autostart` as `0` (or `no`)
   # clusvcadm -Z RESOURCE  <-- whole service/vm only

P: # pcs resource unmanage RESOURCE
   # pcs resource meta RESOURCE is-managed=false



References
==========

[1] http://oss.clusterlabs.org/pipermail/users/2016-January/002197.html

: vim: set ft=rst:  <-- not exactly, but better than nothing