headers
Troy Wolf | 2 Dec 2008 00:21

Log Shipping fails after Switchover

First, a question that I know has been asked before, but after
grepping and perusing months of the maillist archives, I did not find
the answer: Why is it required to use a Subscriber to generate log
files? Why can't the Origin slon process be started with the option to
generate logs? It just so happens that in my case, I have an Origin
and a normal Subscriber and a third node that I replicate via Log
Shipping. However, I imagine lots of people would have a need for a
single Origin and a single, WAN-connected node they want to replicate
via Log Shipping.

And now onto the issue at hand...

PostgreSQL 8.2 on SUSE ES 10
Slony1-1.2.12

The documentation at http://slony.info/documentation/logshipping.html states:
----------------------------------------------------------------------------------------------------------------------------

14.2. What takes place when a FAILOVER/ MOVE SET takes place?

Nothing special. So long as the archiving node remains a subscriber,
it will continue to generate logs.

If the archiving node becomes the origin, on the other hand, it will
continue to generate logs.

----------------------------------------------------------------------------------------------------------------------------
Hmmm..."If the archiving node becomes the origin, on the other hand,
it will continue to generate logs.". Yes, indeed, it continues to
generate logs. Unfortunately, the logs do not include any of the
(Continue reading)

Permalink | Reply | 3 comments |
headers
Jan Wieck | 1 Dec 2008 23:29
Picon
Favicon

Re: Log Shipping fails after Switchover

On 12/1/2008 6:21 PM, Troy Wolf wrote:
> First, a question that I know has been asked before, but after
> grepping and perusing months of the maillist archives, I did not find
> the answer: Why is it required to use a Subscriber to generate log
> files? Why can't the Origin slon process be started with the option to
> generate logs? It just so happens that in my case, I have an Origin
> and a normal Subscriber and a third node that I replicate via Log
> Shipping. However, I imagine lots of people would have a need for a
> single Origin and a single, WAN-connected node they want to replicate
> via Log Shipping.
> 
> And now onto the issue at hand...
> 
> PostgreSQL 8.2 on SUSE ES 10
> Slony1-1.2.12
> 
> The documentation at http://slony.info/documentation/logshipping.html states:
> ----------------------------------------------------------------------------------------------------------------------------
> 
> 14.2. What takes place when a FAILOVER/ MOVE SET takes place?
> 
> Nothing special. So long as the archiving node remains a subscriber,
> it will continue to generate logs.
> 
> If the archiving node becomes the origin, on the other hand, it will
> continue to generate logs.
> 
> ----------------------------------------------------------------------------------------------------------------------------
> Hmmm..."If the archiving node becomes the origin, on the other hand,
> it will continue to generate logs.". Yes, indeed, it continues to
(Continue reading)

Permalink | Reply | 2 comments |
headers
Troy Wolf | 2 Dec 2008 03:54

Re: Log Shipping fails after Switchover

Thanks for the reply, Jan!

On Mon, Dec 1, 2008 at 4:29 PM, Jan Wieck <JanWieck@...> wrote:
> Well, it certainly is a bug that it continues to generate logs ... we will
> see to fix it and stop it from creating useless files.
>

So you confirm the 14.2 documentation is incorrect. That part will be
easy to fix at least! :)

> The reason is the same as for not being able to create such a setup in the
> first place. Log shipping is implemented as a byproduct of the normal
> replication process done by a subscriber. That includes keeping track of
> what has been replicated thus far.

I say this with limited knowledge of how Slony works under the hood,
but it seems obvious that the Slony origin knows about all the changes
because it communicates them to the subscriber, right? In any case,
yes, the ability to generate log files directly from the origin would
be useful to people I think.

Can you speak to Slony II and it's Log Shipping capabilities? Is Log
Shipping improved, enhanced, or otherwise in version 2?
Permalink | Reply | 1 comment |
headers
Jan Wieck | 2 Dec 2008 05:48
Picon
Favicon

Re: Log Shipping fails after Switchover

On 12/1/2008 9:54 PM, Troy Wolf wrote:
> Thanks for the reply, Jan!
> 
> On Mon, Dec 1, 2008 at 4:29 PM, Jan Wieck <JanWieck@...> wrote:
>> Well, it certainly is a bug that it continues to generate logs ... we will
>> see to fix it and stop it from creating useless files.
>>
> 
> So you confirm the 14.2 documentation is incorrect. That part will be
> easy to fix at least! :)
> 
>> The reason is the same as for not being able to create such a setup in the
>> first place. Log shipping is implemented as a byproduct of the normal
>> replication process done by a subscriber. That includes keeping track of
>> what has been replicated thus far.
> 
> I say this with limited knowledge of how Slony works under the hood,
> but it seems obvious that the Slony origin knows about all the changes
> because it communicates them to the subscriber, right? In any case,
> yes, the ability to generate log files directly from the origin would
> be useful to people I think.

The information is all there, that much is true.

The original idea was to create a separate process that "acts" like 
another node based on a "spool node", which the origin never connects 
to. That separate process would then write those logs.

> Can you speak to Slony II and it's Log Shipping capabilities? Is Log
> Shipping improved, enhanced, or otherwise in version 2?
(Continue reading)

Permalink | Reply | 0 comments |
headers
Josh Harrison | 3 Dec 2008 21:21
Picon

1 Slave and 2 different masters ?

Hi,

Can 1 database be a slave for 2 different masters?
ie., For example I have 2 databases - 1 holding "Instittion-A" data and another database for "Institution-B" data.
These 2 database have some common tables with the same table structure.

Is it possible to have a Slony replication from "Institution A"  database to "SLAVE-x"
and a slony replication from "Institution-B" database to "SLAVE-x".
These 2 "Institution" databases wont be accessing the same data...they have their own data but can Slony replicate them in the same slave?

Thanks
Josh

_______________________________________________
Slony1-general mailing list
Slony1-general@...
http://lists.slony.info/mailman/listinfo/slony1-general
Permalink | Reply | 3 comments |
headers
salman | 3 Dec 2008 21:28

Re: 1 Slave and 2 different masters ?

Josh Harrison wrote:
> Hi,
>
> Can 1 database be a slave for 2 different masters?
> ie., For example I have 2 databases - 1 holding "Instittion-A" data and
> another database for "Institution-B" data.
> These 2 database have some common tables with the same table structure.
>
> Is it possible to have a Slony replication from "Institution A"  database to
> "SLAVE-x"
> and a slony replication from "Institution-B" database to "SLAVE-x".
> These 2 "Institution" databases wont be accessing the same data...they have
> their own data but can Slony replicate them in the same slave?
>
> Thanks
> Josh
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Slony1-general mailing list
> Slony1-general@...
> http://lists.slony.info/mailman/listinfo/slony1-general
>   
Maybe by creating two sets with the tables for institution A and B split 
between them and then subscribing the A set to master-1, and the B set 
to master-2 might work?

-salman
Permalink | Reply | 0 comments |
headers
Christopher Browne | 3 Dec 2008 22:21

Re: 1 Slave and 2 different masters ?

"Josh Harrison" <joshques@...> writes:
> Can 1 database be a slave for 2 different masters?
> ie., For example I have 2 databases - 1 holding "Instittion-A" data and another database for
"Institution-B" data.
> These 2 database have some common tables with the same table structure.
> Is it possible to have a Slony replication from "Institution A"  database to "SLAVE-x"
> and a slony replication from "Institution-B" database to "SLAVE-x".
> These 2 "Institution" databases wont be accessing the same data...they have their own data but can Slony
replicate them in the same slave?

No, that'll be a problem.

Slony-I wants to truncate the table at the time of subscription, which
would cause a certain amount of inconvenience - you're certain to lose
one institution's set of data when you subscribe the second one.

What you could do instead, that *would* work, would be one of two
things:

  a) Create two separate tables, and combine their data together, when
     querying, by use of a VIEW.

  b) Create a parent table, and have the two tables as "children" via
     the INHERITS functionality, and get a common view, when you need
     it, via selecting from the parent table.

     http://www.postgresql.org/docs/8.3/static/ddl-partitioning.html
--

-- 
select 'cbbrowne' || ' <at> ' || 'linuxfinances.info';
http://linuxfinances.info/info/spreadsheets.html
"you  can   obvioulsy understand what  i'm  saying.  you're just being
pendantic." -- bazzz777@...
Permalink | Reply | 1 comment |
headers
Filip Rembiałkowski | 4 Dec 2008 12:15
Picon
Gravatar

Re: slon_start and message: slony is already running



2008/11/27 darekk <dariusz.klimas <at> indigo.pl>

hello.


i have a problem with launch slony-I on my linux ( debian ) servers.

when i make:

root <at> gazeta:/var/run/slony1# su - slony
slony <at> gazeta:~$ slon_start --config /etc/slony1/targeo-mobi.conf node2
Slon is already running for the 'replication_targeo_mobi' cluster.

and this message is on master and slave.
the log file for replication_targeo_mobi is not created by slony, and when i
list processes, the process for replication_targeo_mobi does not  exists

someone may have some idea of what may be a problem?


slon tools thinks there's slon process running for this node.

dive into get_pid sub in slon-tools.pm to debug things.




--
Filip Rembiałkowski
_______________________________________________
Slony1-general mailing list
Slony1-general@...
http://lists.slony.info/mailman/listinfo/slony1-general
Permalink | Reply | 0 comments |
headers
Filip Rembiałkowski | 4 Dec 2008 12:20
Picon
Gravatar

Re: Slow Replication on a single cluster.



2008/11/27 René-Etienne Muscat <remuscat <at> gmail.com>
Hi everyone, I have a problem with slony replicating slowly on a particular cluster.
This problem came up after slony replication had stopped for a weekend and this particular cluster (replication set) had a backlog of about 3million rows (about 1.2million on sl_log1 and 1.8million on sl_log2).

with that much of a backlog we often found that it's faster to rebuild slave nodes from scratch than to wait for "normal" catch up.



After resuming slony replication (it had a problem with a table locks), all the replication sets resumed normally (inserting over 15,000 rows per batch), except for this particular set.
When I examined the log for this set on the slave, I saw a lot of SYNC events, but few fetch/delivery events, and the inserts are just about 350rows each time.
I also noted that it was giving a lot of log switch failures ("log switch to sl_log_1 still in progress - sl_log_2 not truncated").
I first tried stopping all replication and starting replication on this problematic set, but still replication was slow (lots of SYNC events but very few fetch/delivery events).
I also tried to resolve the issue by stopping all replication and vacuuming the sl_logs, without any success.
Does anyone know why this is happening? Is this normal? Can I do somthing to speed this up?


which pg version? which slony-I version?
do you have all needed indexes on sl_log* in place?
did you try to REINDEX them? (they are probably "bloated")


I know it's a late reply but HTH.



--
Filip Rembiałkowski
_______________________________________________
Slony1-general mailing list
Slony1-general@...
http://lists.slony.info/mailman/listinfo/slony1-general
Permalink | Reply | 1 comment |
headers
René-Etienne Muscat | 4 Dec 2008 12:47
Picon

Re: Slow Replication on a single cluster.

Filip,
Thanks for the reply, but I solved the issue by doing as you suggested, dropped and recreated them from scratch.

Thanks for your reply,
Rene

On Thu, Dec 4, 2008 at 12:20 PM, Filip Rembiałkowski <filip.rembialkowski <at> gmail.com> wrote:


2008/11/27 René-Etienne Muscat <remuscat <at> gmail.com>

Hi everyone, I have a problem with slony replicating slowly on a particular cluster.
This problem came up after slony replication had stopped for a weekend and this particular cluster (replication set) had a backlog of about 3million rows (about 1.2million on sl_log1 and 1.8million on sl_log2).

with that much of a backlog we often found that it's faster to rebuild slave nodes from scratch than to wait for "normal" catch up.



After resuming slony replication (it had a problem with a table locks), all the replication sets resumed normally (inserting over 15,000 rows per batch), except for this particular set.
When I examined the log for this set on the slave, I saw a lot of SYNC events, but few fetch/delivery events, and the inserts are just about 350rows each time.
I also noted that it was giving a lot of log switch failures ("log switch to sl_log_1 still in progress - sl_log_2 not truncated").
I first tried stopping all replication and starting replication on this problematic set, but still replication was slow (lots of SYNC events but very few fetch/delivery events).
I also tried to resolve the issue by stopping all replication and vacuuming the sl_logs, without any success.
Does anyone know why this is happening? Is this normal? Can I do somthing to speed this up?


which pg version? which slony-I version?
do you have all needed indexes on sl_log* in place?
did you try to REINDEX them? (they are probably "bloated")


I know it's a late reply but HTH.



--
Filip Rembiałkowski

_______________________________________________
Slony1-general mailing list
Slony1-general@...
http://lists.slony.info/mailman/listinfo/slony1-general
Permalink | Reply | 0 comments |

Gmane