<!--  -->
<sect1 id="logshipping">
<title>Log Shipping - &slony1; with Files</title>
<indexterm><primary>log shipping</primary></indexterm>

<para> &slony1; has the ability to serialize the updates to go out into log
files that can be kept in a spool directory.</para>

<para> The spool files could then be transferred via whatever means
was desired to a <quote>slave system,</quote> whether that be via FTP,
rsync, or perhaps even by pushing them onto a 1GB <quote>USB
key</quote> to be sent to the destination by clipping it to the ankle
of some sort of <quote>avian transport</quote> system.</para>

<para> There are plenty of neat things you can do with a data stream
in this form, including:

<itemizedlist>
  
  <listitem><para> Replicating to nodes that
  <emphasis>aren't</emphasis> securable</para></listitem>

  <listitem><para> Replicating to destinations where it is not
  possible to set up bidirection communications</para></listitem>

  <listitem><para> Supporting a different form of <acronym>PITR</acronym>
  (Point In Time Recovery) that filters out read-only transactions and
  updates to tables that are not of interest.</para></listitem>

  <listitem><para> If some disaster strikes, you can look at the logs
  of queries in detail</para>

  <para> This makes log shipping potentially useful even though you
  might not intend to actually create a log-shipped
  node.</para></listitem>

  <listitem><para> This is a really slick scheme for building load for
  doing tests</para></listitem>

  <listitem><para> We have a data <quote>escrow</quote> system that
  would become incredibly cheaper given log shipping</para></listitem>

  <listitem><para> You may apply triggers on the <quote>disconnected
  node </quote> to do additional processing on the data</para>

  <para> For instance, you might take a fairly <quote>stateful</quote>
  database and turn it into a <quote>temporal</quote> one by use of
  triggers that implement the techniques described in
  <citation>Developing Time-Oriented Database Applications in SQL
  </citation> by <ulink url="http://www.cs.arizona.edu/people/rts/">
  Richard T. Snodgrass</ulink>.</para></listitem>

</itemizedlist></para>

<qandaset>
<qandaentry>

<question> <para> Where are the <quote>spool files</quote> for a
subscription set generated?</para>
</question>

<answer> <para> Any <link linkend="slon"> slon </link> subscriber node
can generate them by adding the <option>-a</option> option.</para>

<note><para> Notice that this implies that in order to use log
shipping, you must have at least one subscriber node. </para></note>
</answer>

</qandaentry>

<qandaentry> <question> <para> What takes place when a <xref
linkend="stmtfailover">/ <xref linkend="stmtmoveset"> takes
place?</para></question>

<answer><para> Nothing special.  So long as the archiving node remains
a subscriber, it will continue to generate logs.</para></answer>

<answer> <warning> <para>If the archiving node becomes the origin, on
the other hand, it will continue to generate logs.</para> </warning></answer>
</qandaentry>

<qandaentry> <question> <para> What if we run out of <quote>spool
space</quote>?  </para></question>

<answer><para> The node will stop accepting <command>SYNC</command>s
until this problem is alleviated.  The database being subscribed to
will also fall behind.  </para></answer>
</qandaentry>

<qandaentry>
<question> <para> How do we set up a subscription?  </para></question>

<answer><para> The script in <filename>tools</filename> called
<application>slony1_dump.sh</application> is a shell script that dumps
the <quote>present</quote> state of the subscriber node.</para>

<para> You need to start the <application><link linkend="slon"> slon
</link></application> for the subscriber node with logging turned on.
At any point after that, you can run
<application>slony1_dump.sh</application>, which will pull the state
of that subscriber as of some <command>SYNC</command> event.  Once the
dump completes, all the <command>SYNC</command> logs generated from
the time that dump <emphasis>started</emphasis> may be added to the
dump in order to get a <quote>log shipping subscriber.</quote>
</para></answer>
</qandaentry>

<qandaentry> <question><para> What are the limitations of log
shipping? </para>
</question>

<answer><para> In the initial release, there are rather a lot of
limitations.  As releases progress, hopefully some of these
limitations may be alleviated/eliminated. </para> </answer>

<answer><para> The log shipping functionality amounts to
<quote>sniffing</quote> the data applied at a particular subscriber
node.  As a result, you must have at least one <quote>regular</quote>
node; you cannot have a cluster that consists solely of an origin and
a set of <quote>log shipping nodes.</quote>. </para></answer>

<answer><para> The <quote>log shipping node</quote> tracks the
entirety of the traffic going to a subscriber.  You cannot separate
things out if there are multiple replication sets.  </para></answer>

<answer><para> The <quote>log shipping node</quote> presently only
fully tracks <command>SYNC</command> events.  This should be
sufficient to cope with <emphasis>some</emphasis> changes in cluster
configuration, but not others.  </para>

<para> A number of event types <emphasis> are </emphasis> handled in
such a way that log shipping copes with them:

<itemizedlist>

<listitem><para><command>SYNC </command> events are, of course,
handled.</para></listitem>

<listitem><para><command>DDL_SCRIPT</command> is handled.</para></listitem>

<listitem><para><command> UNSUBSCRIBE_SET </command></para> 

<para> This event, much like <command>SUBSCRIBE_SET</command> is not
handled by the log shipping code.  But its effect is, namely that
<command>SYNC</command> events on the subscriber node will no longer
contain updates to the set.</para>

<para> Similarly, <command>SET_DROP_TABLE</command>,
<command>SET_DROP_SEQUENCE</command>,
<command>SET_MOVE_TABLE</command>,
<command>SET_MOVE_SEQUENCE</command>,
<command>DROP_SET</command>,
<command>MERGE_SET</command>,
<command>SUBSCRIBE_SET</command> will be handled
<quote>apropriately</quote>.</para></listitem>



<listitem><para> The various events involved in node configuration are
irrelevant to log shipping:

<command>STORE_NODE</command>,
<command>ENABLE_NODE</command>,
<command>DROP_NODE</command>,
<command>STORE_PATH</command>,
<command>DROP_PATH</command>,
<command>STORE_LISTEN</command>,
<command>DROP_LISTEN</command></para></listitem>

<listitem><para> Events involved in describing how particular sets are
to be initially configured are similarly irrelevant:

<command>STORE_SET</command>,
<command>SET_ADD_TABLE</command>,
<command>SET_ADD_SEQUENCE</command>,
<command>STORE_TRIGGER</command>,
<command>DROP_TRIGGER</command>,
</para></listitem>

</itemizedlist>
</para>
</answer>

<answer><para> It would be nice to be able to turn a <quote>log
shipped</quote> node into a fully communicating &slony1; node that you
could failover to.  This would be quite useful if you were trying to
construct a cluster of (say) 6 nodes; you could start by creating one
subscriber, and then use log shipping to populate the other 4 in
parallel.</para>

<para> This usage is not supported, but presumably one could take
an application outage and promote the log-shipping node to a normal
slony node with the OMIT COPY option of SUBSCRIBE SET.
 </para></answer>
</qandaentry>
</qandaset>

<sect2><title> Usage Hints </title>

<note> <para> Here are some more-or-less disorganized notes about how
you might want to use log shipping...</para> </note>

<itemizedlist>

<listitem><para> You <emphasis>don't</emphasis> want to blindly apply
<command>SYNC</command> files because any given
<command>SYNC</command> file may <emphasis>not</emphasis> be the right
one.  If it's wrong, then the result will be that the call to
<function> setsyncTracking_offline() </function> will fail, and your
<application> psql</application> session will <command> ABORT
</command>, and then run through the remainder of that
<command>SYNC</command> file looking for a <command>COMMIT</command>
or <command>ROLLBACK</command> so that it can try to move on to the
next transaction.</para>

<para> But we <emphasis> know </emphasis> that the entire remainder of
the file will fail!  It is futile to go through the parsing effort of
reading the remainder of the file.</para>

<para> Better idea: 

<itemizedlist>

<listitem><para> The table, on the log shipped node, tracks which log
it most recently applied in table
<envar>sl_archive_tracking</envar>. </para>

<para> Thus, you may predict the ID number of the next file by taking
the latest counter from this table and adding 1.</para>
</listitem>

<listitem><para> There is still variation as to the filename,
depending on what the overall set of nodes in the cluster are.  All
nodes periodically generate <command>SYNC</command> events, even if
they are not an origin node, and the log shipping system does generate
logs for such events. </para>

<para> As a result, when searching for the next file, it is necessary
to search for files in a manner similar to the following:

<programlisting>
ARCHIVEDIR=/var/spool/slony/archivelogs/node4
SLONYCLUSTER=mycluster
PGDATABASE=logshipdb
PGHOST=logshiphost
NEXTQUERY="select at_counter+1 from \"_${SLONYCLUSTER}\".sl_archive_tracking;"
nextseq=`psql -d ${PGDATABASE} -h ${PGHOST} -A -t -c "${NEXTQUERY}"
filespec=`printf "slony1_log_*_%20d.sql"
for file in `find $ARCHIVEDIR -name "${filespec}"; do
   psql -d ${PGDATABASE} -h ${PGHOST} -f ${file}
done
</programlisting>
</para>
</listitem>

<listitem><para> </para> </listitem>
</itemizedlist>

</sect2>

<sect2><title> <application> find-triggers-to-deactivate.sh
</application> </title>

<indexterm><primary> log shipping - trigger deactivation </primary> </indexterm>

<para> It was once pointed out (<ulink
url="http://www.slony.info/bugzilla/show_bug.cgi?id=19"> Bugzilla bug
#19</ulink>) that the dump of a schema may include triggers and rules
that you may not wish to have running on the log shipped node.</para>

<para> The tool <filename> tools/find-triggers-to-deactivate.sh
</filename> was created to assist with this task.  It may be run
against the node that is to be used as a schema source, and it will
list the rules and triggers present on that node that may, in turn
need to be deactivated.</para>

<para> It includes <function>logtrigger</function> and <function>denyaccess</function>
triggers which will may be left out of the extracted schema, but it is
still worth the Gentle Administrator verifying that such triggers are
kept out of the log shipped replica.</para>

</sect2>
<sect2> <title> <application>slony_logshipper </application> Tool </title>

<indexterm><primary>logshipping: slony_logshipper tool</primary></indexterm>

<para> As of version 1.2.12, &slony1; has a tool designed to help
apply logs, called <application>slony_logshipper</application>.  It is
run with three sorts of parameters:</para>

<itemizedlist>
<listitem><para> Options, chosen from the following: </para> 
<itemizedlist>
<listitem><para><option>h</option> </para> <para>    display this help text and exit </para> </listitem>
<listitem><para><option>v</option> </para> <para>    display program version and exit </para> </listitem>
<listitem><para><option>q</option> </para> <para>    quiet mode </para> </listitem>
<listitem><para><option>l</option> </para> <para>    cause running daemon to reopen its logfile </para> </listitem>
<listitem><para><option>r</option> </para> <para>    cause running daemon to resume after error </para> </listitem>
<listitem><para><option>t</option> </para> <para>    cause running daemon to enter smart shutdown mode </para> </listitem>
<listitem><para><option>T</option> </para> <para>    cause running daemon to enter immediate shutdown mode </para> </listitem>
<listitem><para><option>c</option> </para> <para>    destroy existing semaphore set and message queue            (use with caution) </para> </listitem>
<listitem><para><option>f</option> </para> <para>    stay in foreground (don't daemonize) </para> </listitem>
<listitem><para><option>w</option> </para> <para>    enter smart shutdown mode immediately </para> </listitem>
</itemizedlist>
</listitem>
<listitem><para> A specified log shipper configuration file </para>
<para> This configuration file consists of the following specifications:</para>
<itemizedlist>
<listitem><para> <command>logfile = './offline_logs/logshipper.log';</command></para> 
<para> Where the log shipper will leave messages.</para> </listitem>
<listitem><para> <command>cluster name = 'T1';</command></para> <para> Cluster name </para> </listitem>
<listitem><para> <command>destination database	= 'dbname=slony_test3';</command></para> <para> Optional conninfo for the destination database.  If given, the log shipper will connect to this database, and apply logs to it. </para> </listitem>
<listitem><para> <command>archive dir = './offline_logs';</command></para> <para>The archive directory is required when running in <quote>database-connected</quote> mode to have a place to scan for missing (unapplied) archives. </para> </listitem>
<listitem><para> <command>destination dir = './offline_result';</command></para> <para> If specified, the log shipper will write the results of data massaging into result logfiles in this directory.</para> </listitem>
<listitem><para> <command>max archives = 3600;</command></para> <para> This fights eventual resource leakage; the daemon will enter <quote>smart shutdown</quote> mode automatically after processing this many archives. </para> </listitem>
<listitem><para> <command>ignore table "public"."history";</command></para> <para> One may filter out single tables  from log shipped replication </para> </listitem>
<listitem><para> <command>ignore namespace "public";</command></para> <para> One may filter out entire namespaces  from log shipped replication </para> </listitem>
<listitem><para> <command>rename namespace "public"."history" to "site_001"."history";</command></para> <para> One may rename specific tables.</para> </listitem>
<listitem><para> <command>rename namespace "public" to "site_001";</command></para> <para> One may rename entire namespaces.</para> </listitem>
<listitem><para> <command>post processing command = 'gzip -9 $inarchive';</command></para> <para> Pre- and post-processing commands are executed via <function>system(3)</function>. </para> </listitem>
</itemizedlist>

<para> An <quote>@</quote> as the first character causes the exit code to be ignored.  Otherwise, a nonzero exit code is treated as an error and causes processing to abort. </para>

<para> Pre- and post-processing commands have two further special variables defined: </para>
<itemizedlist>
<listitem><para> <envar>$inarchive</envar>  - indicating incoming archive filename </para> </listitem>
<listitem><para> <envar>$outnarchive</envar>  - indicating outgoing archive filename </para> </listitem>
</itemizedlist>
</listitem>

<listitem><para> <command>error command = ' ( echo
"archive=$inarchive" echo "error messages:" echo "$errortext" ) | mail
-s "Slony log shipping failed" postgres@localhost ';</command></para>

<para>  The error command indicates a command to execute upon encountering an error.  All logging since the last successful completion of an archive is available in the <envar>$errortext</envar> variable. </para> 

<para> In the example shown, this sends an email to the DBAs upon
encountering an error.</para> </listitem>

<listitem><para> Archive File Names</para>

<para> Each filename is added to the SystemV Message queue for
processing by a <application>slony_logshipper</application>
process. </para>

</listitem>

</itemizedlist>

</sect2>
</sect1>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"slony.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->
