= Resource Constraints =

indexterm:[Resource,Constraints]

== Scores ==

Scores of all kinds are integral to how the cluster works.
Practically everything from moving a resource to deciding which
resource to stop in a degraded cluster is achieved by manipulating
scores in some way.

Scores are calculated on a per-resource basis, and any node with a
negative score for a resource can't run that resource.  After
calculating the scores for a resource, the cluster then chooses the
node with the highest one.

=== Infinity Math ===

Pacemaker implements +INFINITY+ (or equivalently, ++INFINITY+) internally as a
score of 1,000,000. Addition and subtraction with it follow these three basic
rules:

* Any value + +INFINITY+ = +INFINITY+
* Any value - +INFINITY+ = +-INFINITY+
* +INFINITY+ - +INFINITY+ = +-INFINITY+

[NOTE]
======
What if you want to use a score higher than 1,000,000? Typically this possibility
arises when someone wants to base the score on some external metric that might
go above 1,000,000.

The short answer is you can't.

The long answer is it is sometimes possible work around this limitation
creatively. You may be able to set the score to some computed value based on
the external metric rather than use the metric directly. For nodes, you can
store the metric as a node attribute, and query the attribute when computing
the score (possibly as part of a custom resource agent).
======

== Deciding Which Nodes a Resource Can Run On ==

indexterm:[Location Constraints]
indexterm:[Resource,Constraints,Location]
There are two alternative strategies for specifying which nodes a
resources can run on.  One way is to say that by default they can run
anywhere and then create location constraints for nodes that are not
allowed.  The other option is to have nodes "opt-in" --  to start with
nothing able to run anywhere and selectively enable allowed nodes.

Whether you should choose opt-in or opt-out depends on your
personal preference and the make-up of your cluster.  If most of your
resources can run on most of the nodes, then an opt-out arrangement is
likely to result in a simpler configuration.  On the other-hand, if
most resources can only run on a small subset of nodes, an opt-in
configuration might be simpler.

=== Location Properties ===

.Properties for Simple Location Constraints
[width="95%",cols="2m,1,5<a",options="header",align="center"]
|=========================================================

|Field
|Default
|Description

|id
|
|A unique name for the constraint
indexterm:[id,Location Constraints]
indexterm:[Constraints,Location,id]

|rsc
|
|A resource name
indexterm:[rsc,Location Constraints]
indexterm:[Constraints,Location,rsc]

|node
|
|A node's name
indexterm:[node,Location Constraints]
indexterm:[Constraints,Location,node]

|score
|
|Positive values indicate the resource should run on this
 node. Negative values indicate the resource should not run on this node.

Values of \+/- +INFINITY+ change "should"/"should not" to "must"/"must not".
indexterm:[score,Location Constraints]
indexterm:[Constraints,Location,score]

|resource-discovery
|always
|Whether Pacemaker should perform resource discovery
on this node for the specified resource. Limiting resource discovery to
a subset of nodes the resource is physically capable of running on
can significantly boost performance when a large set of nodes are present.
When pacemaker_remote is in use to expand the node count into the hundreds of
nodes range, this option should be considered.

* +always:+ Always perform resource discovery for the specified resource on this node.
* +never:+ Never perform resource discovery for the specified resource on this node.
  This option should generally be used with a -INFINITY score, although that is not strictly
  required.
* +exclusive:+ Perform resource discovery for the specified resource only on
  this node (and other nodes similarly marked as exclusive). Multiple location
  constraints using 'exclusive' discovery for the same resource across
  different nodes creates a subset of nodes resource-discovery is exclusive to.
  If a resource is marked for 'exclusive' discovery on one or more nodes, that
  resource is only allowed to be placed within that subset of nodes.

indexterm:[Resource Discovery,Location Constraints]
indexterm:[Constraints,Location,Resource Discovery]

|=========================================================

=== Asymmetrical "Opt-In" Clusters ===
indexterm:[Asymmetrical Opt-In Clusters]
indexterm:[Cluster Type,Asymmetrical Opt-In]

To create an opt-in cluster, start by preventing resources from
running anywhere by default:

----
# crm_attribute --name symmetric-cluster --update false
----

Then start enabling nodes.  The following fragment says that the web
server prefers *sles-1*, the database prefers *sles-2* and both can
fail over to *sles-3* if their most preferred node fails.

.Opt-in location constraints for two resources
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-3" score="0"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-2" score="200"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-3" score="0"/>
</constraints>
-------
======

=== Symmetrical "Opt-Out" Clusters ===
indexterm:[Symmetrical Opt-Out Clusters]
indexterm:[Cluster Type,Symmetrical Opt-Out]

To create an opt-out cluster, start by allowing resources to run
anywhere by default:

----
# crm_attribute --name symmetric-cluster --update true
----

Then start disabling nodes.  The following fragment is the equivalent
of the above opt-in configuration.

.Opt-out location constraints for two resources
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2-dont-run" rsc="Webserver" node="sles-2" score="-INFINITY"/>
    <rsc_location id="loc-3-dont-run" rsc="Database" node="sles-1" score="-INFINITY"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="200"/>
</constraints>
-------
======

[[node-score-equal]]
=== What if Two Nodes Have the Same Score ===

If two nodes have the same score, then the cluster will choose one.
This choice may seem random and may not be what was intended, however
the cluster was not given enough information to know any better.

.Constraints where a resource prefers two nodes equally
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="INFINITY"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-2" score="INFINITY"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-1" score="500"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="300"/>
    <rsc_location id="loc-5" rsc="Database" node="sles-2" score="200"/>
</constraints>
-------
======

In the example above, assuming no other constraints and an inactive
cluster, +Webserver+ would probably be placed on +sles-1+ and +Database+ on
+sles-2+.  It would likely have placed +Webserver+ based on the node's
uname and +Database+ based on the desire to spread the resource load
evenly across the cluster.  However other factors can also be involved
in more complex configurations.

[[s-resource-ordering]]
== Specifying the Order in which Resources Should Start/Stop ==

indexterm:[Resource,Constraints,Ordering]
indexterm:[Resource,Start Order]
indexterm:[Ordering Constraints]

The way to specify the order in which resources should start is by
creating +rsc_order+ constraints.

[IMPORTANT]
====
Ordering constraints affect 'only' the ordering of resources;
they do not require that the resources be placed on the
same node. If you want resources to be started on the same node
'and' in a specific order, you need an ordering constraint 'and'
a location constraint (see <<s-resource-colocation>>).
====

=== Ordering Properties ===

.Properties of an Ordering Constraint
[width="95%",cols="1m,1,4<a",options="header",align="center"]
|=========================================================

|Field
|Default
|Description

|id
|
|A unique name for the constraint
indexterm:[id,Ordering Constraints]
indexterm:[Constraints,Ordering,id]

|first
|
|Name of the resource that the +then+ resource depends on
indexterm:[first,Ordering Constraints]
indexterm:[Constraints,Ordering,first]

|then
|
|Name of the dependent resource
indexterm:[then,Ordering Constraints]
indexterm:[Constraints,Ordering,then]

|first-action
|start
|The action that the +first+ resource must complete before +then-action+
 can be initiated for the +then+ resource.  Allowed values: +start+,
 +stop+, +promote+, +demote+.
 indexterm:[first-action,Ordering Constraints]
 indexterm:[Constraints,Ordering,first-action]

|then-action
|value of +first-action+
|The action that the +then+ resource can execute only after the
 +first-action+ on the +first+ resource has completed.  Allowed
 values: +start+, +stop+, +promote+, +demote+.
 indexterm:[then-action,Ordering Constraints]
 indexterm:[Constraints,Ordering,then-action]

|kind
|
|How to enforce the constraint. Allowed values:

* +Optional:+ Just a suggestion. Only applies if both resources are
  executing the specified actions. Any change in state by the +first+ resource
  will have no effect on the +then+ resource.
* +Mandatory:+ Always. If +first+ does not perform +first-action+, +then+ will
  not be allowed to performed +then-action+. If +first+ is restarted, +then+
  (if running) will be stopped beforehand and started afterward.
* +Serialize:+ Ensure that no two stop/start actions occur concurrently
  for the resources. +First+ and +then+ can start in either order,
  but one must complete starting before the other can be started. A typical use
  case is when resource start-up puts a high load on the host.

indexterm:[kind,Ordering Constraints]
indexterm:[Constraints,Ordering,kind]

|symmetrical
|TRUE
|If true, the reverse of the constraint applies for the opposite action (for
 example, if B starts after A starts, then B stops before A stops).
indexterm:[symmetrical,Ordering Constraints]
indexterm:[Ordering Constraints,symmetrical]

|=========================================================

+Promote+ and +demote+ apply to the master role of
<<s-resource-multistate,multi-state>> resources.

=== Optional and mandatory ordering ===

Here is an example of ordering constraints where +Database+ 'must' start before
+Webserver+, and +IP+ 'should' start before +Webserver+ if they both need to be
started:

.Optional and mandatory ordering constraints
======
[source,XML]
-------
<constraints>
<rsc_order id="order-1" first="IP" then="Webserver" kind="Optional"/>
<rsc_order id="order-2" first="Database" then="Webserver" kind="Mandatory" />
</constraints>
-------
======

Because the above example lets +symmetrical+ default to TRUE, 
+Webserver+ must be stopped before +Database+ can be stopped,
and +Webserver+ should be stopped before +IP+
if they both need to be stopped.

[[s-resource-colocation]]
== Placing Resources Relative to other Resources ==

indexterm:[Resource,Constraints,Colocation]
indexterm:[Resource,Location Relative to other Resources]
When the location of one resource depends on the location of another
one, we call this colocation.

Colocation has an important side-effect: it affects the order in which
resources are assigned to a node.
footnote:['Not' the order in which they are started. For that, see
<<s-resource-ordering>>.]
Think about it: You can't place A relative to B unless you know where B is.
footnote:[
While the human brain is sophisticated enough to read the constraint
in any order and choose the correct one depending on the situation,
the cluster is not quite so smart. Yet.
]

So when you are creating colocation constraints, it is important to
consider whether you should colocate A with B, or B with A.

Another thing to keep in mind is that, assuming A is colocated with
B, the cluster will take into account A's preferences when
deciding which node to choose for B.

For a detailed look at exactly how this occurs, see
http://clusterlabs.org/doc/Colocation_Explained.pdf[Colocation Explained].

=== Colocation Properties ===

.Properties of a Colocation Constraint
[width="95%",cols="2m,5<",options="header",align="center"]
|=========================================================

|Field
|Description

|id
|A unique name for the constraint.
indexterm:[id,Colocation Constraints]
indexterm:[Constraints,Colocation,id]

|rsc
|The name of a resource that should be located relative to +with-rsc+.
indexterm:[rsc,Colocation Constraints]
indexterm:[Constraints,Colocation,rsc]

|with-rsc
|The name of the resource used as the colocation target. The cluster will
decide where to put this resource first and then decide where to put +rsc+.
 indexterm:[with-rsc,Colocation Constraints]
 indexterm:[Constraints,Colocation,with-rsc]

|score
|Positive values indicate the resources should run on the same
 node. Negative values indicate the resources should run on
 different nodes. Values of \+/- +INFINITY+ change "should" to "must".
 indexterm:[score,Colocation Constraints]
 indexterm:[Constraints,Colocation,score]

|=========================================================

=== Mandatory Placement ===

Mandatory placement occurs when the constraint's score is
++INFINITY+ or +-INFINITY+.  In such cases, if the constraint can't be
satisfied, then the +rsc+ resource is not permitted to run.  For
+score=INFINITY+, this includes cases where the +with-rsc+ resource is
not active.

If you need resource +A+ to always run on the same machine as
resource +B+, you would add the following constraint:

.Mandatory colocation constraint for two resources
====
[source,XML]
<rsc_colocation id="colocate" rsc="A" with-rsc="B" score="INFINITY"/>
====

Remember, because +INFINITY+ was used, if +B+ can't run on any
of the cluster nodes (for whatever reason) then +A+ will not
be allowed to run. Whether +A+ is running or not has no effect on +B+.

Alternatively, you may want the opposite -- that +A+ 'cannot'
run on the same machine as +B+.  In this case, use
+score="-INFINITY"+.

.Mandatory anti-colocation constraint for two resources
====
[source,XML]
<rsc_colocation id="anti-colocate" rsc="A" with-rsc="B" score="-INFINITY"/>
====

Again, by specifying +-INFINITY+, the constraint is binding.  So if the
only place left to run is where +B+ already is, then
+A+ may not run anywhere.

As with +INFINITY+, +B+ can run even if +A+ is stopped.
However, in this case +A+ also can run if +B+ is stopped, because it still
meets the constraint of +A+ and +B+ not running on the same node.

=== Advisory Placement ===

If mandatory placement is about "must" and "must not", then advisory
placement is the "I'd prefer if" alternative.  For constraints with
scores greater than +-INFINITY+ and less than +INFINITY+, the cluster
will try to accommodate your wishes but may ignore them if the
alternative is to stop some of the cluster resources.

As in life, where if enough people prefer something it effectively
becomes mandatory, advisory colocation constraints can combine with
other elements of the configuration to behave as if they were
mandatory.

.Advisory colocation constraint for two resources
====
[source,XML]
<rsc_colocation id="colocate-maybe" rsc="A" with-rsc="B" score="500"/>
====

[[s-resource-sets-ordering]]
== Ordering Sets of Resources ==

A common situation is for an administrator to create a chain of
ordered resources, such as:

.A chain of ordered resources
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1" first="A" then="B" />
    <rsc_order id="order-2" first="B" then="C" />
    <rsc_order id="order-3" first="C" then="D" />
</constraints>
-------
======

.Visual representation of the four resources' start order for the above constraints
image::images/resource-set.png["Ordered set",width="16cm",height="2.5cm",align="center"]

=== Ordered Set ===

To simplify this situation, there is an alternate format for ordering
constraints:

.A chain of ordered resources expressed as a set
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======

[WARNING]
=========
Always pay attention to how your tools expose this functionality.
In some tools +create set A B+ is *NOT* equivalent to +create A then B+.
=========

While the set-based format is not less verbose, it is significantly
easier to get right and maintain.  It can also be expanded to allow
ordered sets of (un)ordered resources.  In the example below, +A+
and +B+ can both start in parallel, as can +C+ and +D+,
however +C+ and +D+ can only start once _both_ +A+ _and_
 +B+ are active.

.Ordered sets of unordered resources
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
  </constraints>
-------
======

.Visual representation of the start order for two ordered sets of unordered resources
image::images/two-sets.png["Two ordered sets",width="13cm",height="7.5cm",align="center"]

Of course either set -- or both sets -- of resources can also be
internally ordered (by setting +sequential="true"+) and there is no
limit to the number of sets that can be specified.

.Advanced use of set ordering - Three ordered sets, two of which are internally unordered
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="true">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
      <resource_set id="ordered-set-3" sequential="false">
        <resource_ref id="E"/>
        <resource_ref id="F"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======

.Visual representation of the start order for the three sets defined above
image::images/three-sets.png["Three ordered sets",width="16cm",height="7.5cm",align="center"]


=== Resource Set OR Logic ===

The unordered set logic discussed so far has all been "AND" logic.
To illustrate this take the 3 resource set figure in the previous section.
Those sets can be expressed, +(A and B) then \(C) then (D) then (E and F)+.

Say for example we want to change the first set, +(A and B)+, to use "OR" logic
so the sets look like this: +(A or B) then \(C) then (D) then (E and F)+.
This functionality can be achieved through the use of the +require-all+
option.  This option defaults to TRUE which is why the
"AND" logic is used by default.  Setting +require-all=false+ means only one
resource in the set needs to be started before continuing on to the next set.

Note that the +require-all=false+ option only makes sense to use in conjunction
with unordered sets, +sequential=false+.  Think of it like this, +sequential=false+
modifies the set to be an unordered set that uses "AND" logic by default, by adding
+require-all=false+ the unordered set's "AND" logic is flipped to "OR" logic.

.Resource Set "OR" logic: Three ordered sets, where the first set is internally unordered with "OR" logic
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false" require-all="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="true">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
      <resource_set id="ordered-set-3" sequential="false">
        <resource_ref id="E"/>
        <resource_ref id="F"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======


[[s-resource-sets-colocation]]
== Colocating Sets of Resources ==

Another common situation is for an administrator to create a set of
colocated resources.

One way to do this would be to define a resource group (see
<<group-resources>>), but that cannot always accurately express the desired
state.

Another way would be to define each relationship as an individual constraint,
but that causes a constraint explosion as the number of resources and
combinations grow. An example of this approach:

.Chain of colocated resources
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" rsc="A" with-rsc="B" score="INFINITY"/>
    <rsc_colocation id="coloc-2" rsc="B" with-rsc="C" score="INFINITY"/>
    <rsc_colocation id="coloc-3" rsc="C" with-rsc="D" score="INFINITY"/>
</constraints>
-------
======

To make things easier, we allow an alternate form of colocation
constraints using +resource_set+. As with the chained version, a
resource that can't be active prevents any resource that must be
colocated with it from being active.  For example, if +C+ is not
able to run, then both +B+ and by inference +A+ must also remain
stopped. Here is an example +resource_set+:

.Equivalent colocation chain expressed using +resource_set+
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="colocated-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======

[WARNING]
=========
Always pay attention to how your tools expose this functionality.
In some tools +create set A B+ is 'not' equivalent to +create A with B+.
=========

This notation can also be used to tell the cluster
that a set of resources must all be located with a common peer, but
have no dependencies on each other. In this scenario, unlike the
previous, +B+ 'would' be allowed to remain active even if +A+ or +C+ (or
both) were inactive.

.Using colocation sets to specify a common peer
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="colocated-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
      </resource_set>
      <resource_set id="colocated-set-2" sequential="true">
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======

There is no inherent limit to the number and size of the sets used.
The only thing that matters is that in order for any member of one set
in the constraint to be active, all members of sets listed after it must also
be active (and naturally on the same node); and if a set has +sequential="true"+,
then in order for one member of that set to be active, all members listed after it
must also be active. You can even specify the role in which the members of a set
must be in using the set's +role+ attribute.

.A colocation chain where the members of the middle set have no interdependencies and the last has master status.
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="colocated-set-1" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="colocated-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
        <resource_ref id="E"/>
      </resource_set>
      <resource_set id="colocated-set-3" sequential="true" role="Master">
        <resource_ref id="F"/>
        <resource_ref id="G"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======

.Visual representation of a colocation chain where the members of the middle set have no inter-dependencies
image::images/three-sets-complex.png["Colocation chain",width="16cm",height="9cm",align="center"]
