Discussion:
ADAM Replication - 1 instance off issue
(too old to reply)
chicabow
2006-02-08 14:42:54 UTC
Permalink
I have 2 instances runing and replication is happening between the 2
servers. But when I try turn off the first instance and try to connect
to instance 2 via adsi edit I get this error.
'A referral was returned by the server' I thought the point of this
replication is if 1 server fails I can still connect to the second one.
I can connect to the configuration portion of it, but I cannot seem to
get to the instance where the objects are actually stored, the only way
I can is if the first instance is on.
Am I missing something.

Looking for some assistance.

Thanks
Lee Flight
2006-02-08 16:52:42 UTC
Permalink
Hi

you would see that if the replication to instance 2 had not completed.
Did you add the replica of your naming context to instance2 at the
time you installed instance2 or at a later time? How long did you
wait for replication to take place.?

Try running:

repadmin /showrepl <AdamserverName>:<AdamPort>

with AdamserverName the name of the instance1 and then instance2
servers, to check the status of replication.


Lee Flight
Post by chicabow
I have 2 instances runing and replication is happening between the 2
servers. But when I try turn off the first instance and try to connect
to instance 2 via adsi edit I get this error.
'A referral was returned by the server' I thought the point of this
replication is if 1 server fails I can still connect to the second one.
I can connect to the configuration portion of it, but I cannot seem to
get to the instance where the objects are actually stored, the only way
I can is if the first instance is on.
Am I missing something.
Looking for some assistance.
Thanks
chicabow
2006-02-08 19:03:59 UTC
Permalink
Thanks for the quick response.
Yes I did do the replication at the time of installation for instance
2. I have waited a long time more than 30 minutes, even restarted the
service on both servers.

As for the command I tried to run it but am getting errors, what is the
exact syntax. Below is what I use to connect in adsiedit

CN=Main,DC=CData,DC=CA

Do I need to include server name or something? I tried the line you put
above.
C:\WINDOWS\ADAM>repadmin /showrepl main:1234

And I get the following error

C:\WINDOWS\ADAM>repadmin /showrepl main:1234
Repadmin can't connect to a "home server", because of the following
error. Try
specifying a different
home server with /homeserver:[dns name]
Error: An LDAP lookup operation failed with the following error:

LDAP Error 81(0x51): Server Down
Server Win32 Error 0(0x0):
Extended Information:

I cant connect to it via adsi edit so I dont know why it would say
server is down. Please clarify exact syntax and any more information I
may be missing.
chicabow
2006-02-08 19:07:55 UTC
Permalink
Just wanted to correct the last statement, I CAN connect to it via
ADSI, so not sure why i am getting this error.
Lee Flight
2006-02-08 20:29:10 UTC
Permalink
Hi

repadmin /showrepl <servername>:<adam ldap port>

e.g.

repadmin /showrepl myserver1.net:389

Lee Flight
Post by chicabow
Thanks for the quick response.
Yes I did do the replication at the time of installation for instance
2. I have waited a long time more than 30 minutes, even restarted the
service on both servers.
As for the command I tried to run it but am getting errors, what is the
exact syntax. Below is what I use to connect in adsiedit
CN=Main,DC=CData,DC=CA
Do I need to include server name or something? I tried the line you put
above.
C:\WINDOWS\ADAM>repadmin /showrepl main:1234
And I get the following error
C:\WINDOWS\ADAM>repadmin /showrepl main:1234
Repadmin can't connect to a "home server", because of the following
error. Try
specifying a different
home server with /homeserver:[dns name]
LDAP Error 81(0x51): Server Down
I cant connect to it via adsi edit so I dont know why it would say
server is down. Please clarify exact syntax and any more information I
may be missing.
chicabow
2006-02-08 21:00:04 UTC
Permalink
I ran this command on the server server within the ADAM Tools Command
Prompt

C:\WINDOWS\ADAM>repadmin /showrepl cms2:1234
Default-First-Site-Name\CMS2$Main
[e:\nt_adam\ds\ds\src\util\repadmin\repinfo.c, 548] LDAP error 32 (No
Such Objec
t) Win32 Err 2.

Unfortunately I have no idea what this means....I hope you can shed
some light.
I await to hear from you.
Thanks.
Lee Flight
2006-02-08 21:10:26 UTC
Permalink
Hi

what did the command show when run against the
other (original) server in the config set?

Lee Flight
Post by chicabow
I ran this command on the server server within the ADAM Tools Command
Prompt
C:\WINDOWS\ADAM>repadmin /showrepl cms2:1234
Default-First-Site-Name\CMS2$Main
[e:\nt_adam\ds\ds\src\util\repadmin\repinfo.c, 548] LDAP error 32 (No
Such Objec
t) Win32 Err 2.
Unfortunately I have no idea what this means....I hope you can shed
some light.
I await to hear from you.
Thanks.
chicabow
2006-02-08 21:15:24 UTC
Permalink
Looks like its the pretty much same message.

C:\WINDOWS\ADAM>repadmin /showrepl cms1:1234
Default-First-Site-Name\CMS1$Main
[e:\nt_adam\ds\ds\src\util\repadmin\repinfo.c, 548] LDAP error 32 (No
Such Objec
t) Win32 Err 2.


Any ideas?
Lee Flight
2006-02-08 21:32:32 UTC
Permalink
Hi

could you run the command using a account that has administrator
access to the ADAM instance?

Lee Flight
Post by chicabow
Looks like its the pretty much same message.
C:\WINDOWS\ADAM>repadmin /showrepl cms1:1234
Default-First-Site-Name\CMS1$Main
[e:\nt_adam\ds\ds\src\util\repadmin\repinfo.c, 548] LDAP error 32 (No
Such Objec
t) Win32 Err 2.
Any ideas?
chicabow
2006-02-08 21:59:48 UTC
Permalink
This is what I got when I ran it on the second instance. Something I
want to mention as well, below i see references to instances in which
at one point I replicated (so I could get a copy of that instance onto
a server). But now I no longer need them replicating or even there to
begin with.
They are cms-silver and gold, those were used before but no longer
needed.
All I want is, cms1 to replicate to cms2. Which looks like it did and
does, but went I shut off cms1, I cannot connect to cms2. I hope this
makes sense. I will post the results of this command line when I run it
on the first server instance (cms1)

C:\WINDOWS\ADAM>repadmin /showrepl cms2:1234
Default-First-Site-Name\CMS2$Main
DSA Options: (none)
Site Options: (none)
DSA object GUID: 96f8de76-bbee-4492-a793-4ec9dccec89f
DSA invocationID: b73f9b62-2fea-4673-afe0-e47572c2504f


Source: Default-First-Site-Name\CMS-SILVER$Main
******* 33 CONSECUTIVE FAILURES since 2006-02-08 08:39:03
Last error: 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.

Naming Context:
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\CMS-SILVER$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context:
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\CMS-SILVER$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Main,DC=CData,DC=CA
Source: Default-First-Site-Name\CMS-SILVER$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Source: Default-First-Site-Name\GOLD$Main
******* 33 CONSECUTIVE FAILURES since 2006-02-08 08:38:57
Last error: 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.

Naming Context:
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context:
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Main,DC=CData,DC=CA
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Source: Default-First-Site-Name\CMS1$Main
******* 25 CONSECUTIVE FAILURES since 2006-02-08 10:41:27
Last error: 5 (0x5):
Access is denied.

Naming Context:
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\CMS1$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context:
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\CMS1$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Main,DC=CData,DC=CA
Source: Default-First-Site-Name\CMS1$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
chicabow
2006-02-08 22:07:50 UTC
Permalink
Here is the results from instance1 server (cms1)

C:\WINDOWS\ADAM>repadmin /showrepl cms1:1234
Default-First-Site-Name\CMS1$Main
DSA Options: (none)
Site Options: (none)
DSA object GUID: 68e11a3b-b41e-49f1-99dc-e3a2106b37e6
DSA invocationID: 8794faf1-3fac-41aa-8407-1b61b12d8846

==== INBOUND NEIGHBORS ======================================

CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
Last attempt @ 2006-02-08 16:48:24 failed, result 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
658 consecutive failure(s).
Last success @ 2006-01-12 15:44:10.

CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
Last attempt @ 2006-02-08 16:48:45 failed, result 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
658 consecutive failure(s).
Last success @ 2006-01-12 15:44:10.

CN=Main,DC=CData,DC=CA
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
Last attempt @ 2006-02-08 16:49:06 failed, result 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
12 consecutive failure(s).
Last success @ 2006-01-12 15:48:35.

Source: Default-First-Site-Name\CMS2$Main
******* 1 CONSECUTIVE FAILURES since 2006-02-08 16:52:40
Last error: 5 (0x5):
Access is denied.

Naming Context:
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\CMS2$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context:
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\CMS2$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Source: Default-First-Site-Name\CMS-SILVER$Main
******* 658 CONSECUTIVE FAILURES since 2006-01-12 15:48:35
Last error: 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.

Source: Default-First-Site-Name\GOLD$Main
******* 1 CONSECUTIVE FAILURES since 2006-02-08 16:52:35
Last error: 1772 (0x6ec):
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.

Naming Context:
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context:
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Main,DC=CData,DC=CA
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.


I look forward in hearing your feedback and suggestion/solution.
Lee Flight
2006-02-09 10:24:53 UTC
Permalink
Hi

a few things:

if the replicas at cms-silver and cms-gold were no longer required you
really should
have removed them from the replica set. It looks like they are actually no
longer
required in the config set and so you should have uninstalled the ADAM
instance
from them whilst they were still visible to the source server that would
have cleaned
up the connections (which are now all broken).

did the connections that you set up from cms-silver and cms-gold replicate
correctly
and if so what are you doing that is different for the new replication
partner? Are you
using the same ADAM service account? Are the machines members of the same
domain?

The
WARNING: KCC could not add this REPLICA LINK due to error
and
Access is denied

errors both need investigation. If the machines are using Kerberos for
mutual
authentication then "Access is denied" could be a problem with machine SPNs,
account status or even just clock-skew between the machines. For more detail
on the
KCC problem you would need to check the ADAM instance event logs and look
at KCC source messages when you added the second instance, DNS/name
resolution
can be a common problem here (changing machines names or domain membership
can also cause problems)

If this ADAM setup is in production then you should probably open an
incident
with Microsoft to get help on cleaning up the metadata from the replicas
that you
have removed and diagnosing the problem further.


Lee Flight
Post by chicabow
Here is the results from instance1 server (cms1)
C:\WINDOWS\ADAM>repadmin /showrepl cms1:1234
Default-First-Site-Name\CMS1$Main
DSA Options: (none)
Site Options: (none)
DSA object GUID: 68e11a3b-b41e-49f1-99dc-e3a2106b37e6
DSA invocationID: 8794faf1-3fac-41aa-8407-1b61b12d8846
==== INBOUND NEIGHBORS ======================================
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
658 consecutive failure(s).
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
658 consecutive failure(s).
CN=Main,DC=CData,DC=CA
Default-First-Site-Name\CMS-SILVER$Main via RPC
DSA object GUID: 1207e065-f5d5-4f78-9838-43a92654efdb
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
12 consecutive failure(s).
Source: Default-First-Site-Name\CMS2$Main
******* 1 CONSECUTIVE FAILURES since 2006-02-08 16:52:40
Access is denied.
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\CMS2$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\CMS2$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
Source: Default-First-Site-Name\CMS-SILVER$Main
******* 658 CONSECUTIVE FAILURES since 2006-01-12 15:48:35
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
Source: Default-First-Site-Name\GOLD$Main
******* 1 CONSECUTIVE FAILURES since 2006-02-08 16:52:35
The list of RPC servers available for the binding of auto
handles ha
s been exhausted.
CN=Schema,CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D
7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
CN=Configuration,CN={9A366D81-5868-4A0C-87EF-6BE003A8D7CB}
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
Naming Context: CN=Main,DC=CData,DC=CA
Source: Default-First-Site-Name\GOLD$Main
******* WARNING: KCC could not add this REPLICA LINK due to error.
I look forward in hearing your feedback and suggestion/solution.
chicabow
2006-02-09 13:38:09 UTC
Permalink
Thanks for the response, well at some point I did do replication from
one server to another (just to get the instance into another server).
From cms-silver to cms1. Now i longer need cms-silver and want
replication to happen between cms1 and cms2. So there is definitely
metadata that does no longer need to be there.
Something I have always wondered, if I replicate from cms-silver to
cms1. Then I no longer need cms-silver or it gets removed from the
network or something like I want to no longer reference. Can't I just
'stop the replication' from happeneing and treat cms1 as the master.
Which is what I did, and from there i replicated from cms1 to cms2.

As for resolving this issue. I wondered is an option perhaps exporting
or making a backup of the instance Main somewhere, then removing all
the instances completely and starting over but still have a copy of the
exported file and do this again?

Looking forward to your feedback.
chicabow
2006-02-09 14:37:44 UTC
Permalink
I just want to note that, it seems as though replication still happens
between cms1 and cms2. I was connected to cms1 and removed some
entries, and the later disappeared when i connected to cms2. Then I did
the reverse, I connected to cms2 and removed some there, and then
connected to cms1 and they were gone as well.
Should this be happening given the errors from above?
chicabow
2006-02-09 15:02:59 UTC
Permalink
I just want to note that, it seems as though replication still happens
between cms1 and cms2. I was connected to cms1 and removed some
entries, and the later disappeared when i connected to cms2. Then I did
the reverse, I connected to cms2 and removed some there, and then
connected to cms1 and they were gone as well.
Should this be happening given the errors from above?
Lee Flight
2006-02-09 23:34:00 UTC
Permalink
Hi

how are you looking at the directories. If you are using a tool
like ADSIedit that follows LDAP referrals automatically
it could be that one of the servers does not have a replica of the
data and you are just seeing the result of the LDAP referral.
ADSIedit can be misleading in situations where you do not have
replication working.

The way to check is to bind to each directory using a tool like ldp.exe
and see what naming contexts are present or to use repadmin to dump
an object from each directory e.g.

repadmin /showattr cms1:1234 CN=Main,DC=CData,DC=CA

and again for cms2. If you get a referral back from either query
the naming context is not a fit replica on that server.

Lee Flight
chicabow
2006-02-10 16:50:04 UTC
Permalink
I did get a referral back from the second one of cms2. So if I was to
delete the instance on cms2 and tried to re-create it, would that be a
better option? If so, should I replicate from cms1 or export/import
using the ldf file?

Thanks for your help thus far, very appreciated.
Lee Flight
2006-02-10 17:10:56 UTC
Permalink
Hi

uninstalling the instance on cms2 and then re-creating might tell you
something
about the state of the configuration set. Make sure you have a current
backup
of cms1 before doing anything. Assuming the uninstall of cms2 goes OK, then
you could watch the ADAM instance event log for KCC events after
the re-install to see if the replication connections get built.

Export using ldif is usually a last resort if you have security principals
(users)
in your ADAM instance as you will lose their password info.

Lee Flight
Post by chicabow
I did get a referral back from the second one of cms2. So if I was to
delete the instance on cms2 and tried to re-create it, would that be a
better option? If so, should I replicate from cms1 or export/import
using the ldf file?
Thanks for your help thus far, very appreciated.
chicabow
2006-02-11 04:24:19 UTC
Permalink
Just so I know what you mean, exactly what do you consider a backup?
(using what tool and steps).
So if I do this activity on cms2, uninstall and re-install/replicate,
this does not do anything as far as removing the metatdata about
cms-silver and gold right? Or how can I stop the scheduling/replication
of the other items. When doing the scheduling to continue to replicate
every 15 minutes, is there a way to update the schedule so the
replication only happens for a specific instance as opposed ot the
other ones that 'are still in there'.
Thanks.
Lee Flight
2006-02-12 23:24:05 UTC
Permalink
Hi

inline below...
Post by chicabow
Just so I know what you mean, exactly what do you consider a backup?
(using what tool and steps).
See the ADAM help file for backing up an ADAM instance.
Post by chicabow
So if I do this activity on cms2, uninstall and re-install/replicate,
this does not do anything as far as removing the metatdata about
cms-silver and gold right? Or how can I stop the scheduling/replication
of the other items. When doing the scheduling to continue to replicate
every 15 minutes, is there a way to update the schedule so the
replication only happens for a specific instance as opposed ot the
other ones that 'are still in there'.
Currently I cannot offer any advice on how to clean up the links
for the other instances (cms-silver,cms-gold) as I cannot find anything
documented and my attempts to develop a procedure with the standard
tools have failed so far. As I mentioned before if you have this data in
production then you might want to consider opening an incident with
Microsoft to help clean up your configuration set.

Lee Flight
Post by chicabow
Thanks.
Lee Flight
2006-02-09 23:24:10 UTC
Permalink
Hi

inline below...
Post by chicabow
Something I have always wondered, if I replicate from cms-silver to
cms1. Then I no longer need cms-silver or it gets removed from the
network or something like I want to no longer reference. Can't I just
'stop the replication' from happeneing and treat cms1 as the master.
Which is what I did, and from there i replicated from cms1 to cms2.
If you want you take a server out of a configuration set the way to do it is
to uninstall the ADAM instance from that server, that will clean up the
metadata
and connections.
Post by chicabow
As for resolving this issue. I wondered is an option perhaps exporting
or making a backup of the instance Main somewhere, then removing all
the instances completely and starting over but still have a copy of the
exported file and do this again?
I think taking a backup and restoring will probably not give that much of a
clean up as the broken connections will be restored. One possibility would
be to export all of the data using something like ldifde. I do not know of
any guidance for cleaning up metadata in ADAM config sets and my attempts
to do a clean up using dsmgmt in following this thread have not been
successful,
but that might just be my lack of understanding. Again talking to Microsoft
PSS would probably be the way to go if this is production data.

Lee Flight
Post by chicabow
Looking forward to your feedback.
Loading...