Reverting from a schema change crash

[Ib Schrader]

Hi

I've tried locating documents or articles that describe how you recover from
a schema change gone bad. I know this is a very serious situation, but I was
hoping there at least was some way of getting back.

I've heard you can keep a DC offline for the upgrade and if everything fails
use that one DC to rebuild the domain, would that work? because I've also
heard that all DC's have to be online for AD to even accept the schema
change.

Is there any way at all to back up the schema and reload it should it become
corrupted in some way?

Thanks for any input.
Ib

[Ib Schrader]

I was thinking of some kind of corruption in the schema so that "nothing"
works.

Is it possible to recover from that, or is it impossible for the schema to
come in a unworkable state?

[Paul Bergson [MVP-DS]]

Well I hope before you try this in your production system you have a test
system to test this out in. Can you completely break AD with a Schema
Extension? Sure you can, so you better know what you are doing before you
just jump in and start playing. Virtual server is free now, so you don't
need nearly the hardware you used to.

If I recall correctly you can't extend the schema if a DC isn't online, AD
wants to make sure that replication us sync'd up before you start messing
with it.

There are three good rules for projects before you go live

Test
Test
Test
--
Paul Bergson
MVP - Directory Services
MCT, MCSE, MCSA, Security+, BS CSci
2003, 2000 (Early Achiever), NT

http://www.pbbergs.com

Please no e-mails, any questions should be posted in the NewsGroup
This posting is provided "AS IS" with no warranties, and confers no rights.

Post by Ib Schrader
I was thinking of some kind of corruption in the schema so that "nothing"
works.
Is it possible to recover from that, or is it impossible for the schema to
come in a unworkable state?

[Joe Richards [MVP]]

All DCs do not need to be online for a schema change, you can have some
offline backups. However, it won't be a matter of turning them on and
the good stuff replicates back out. You have to kill every DC that is
screwed up and then rebuild them all and repromote them. However that is
your one way to get rolled back. Well you don't have to have a DC
offline, you could whack all DCs and then restore one for every domain
and start out that way. Offline is better for me. Even better and what I
usually do is set up a "lifeboat". I build some a DC for each domain,
move them onto an isolated network. Whack all of the other DCs out of
the config and make sure the test environment works fine, then promote
one or two more test DCs in the isolated environment and make sure that
is all replicating fine. Then I whack the test DCs out of the production
environment and do the schema update. It is actually a good way to test
the schema mods as well to see how they will impact a real copy of the
DIT. This can also be done a little cleaner with virtual DCs (say from a
lag site) but you need to be extra careful since the two sets of DCs in
the different environments can still replicate with each other since you
aren't clearing out the entries for them in the two now separate
environments.

All that being said, I have yet to have heard of a schema change that
blew up a forest. The vast majority of issues are people who wrote their
own schema changes (or used a clueless vendor - And Cisco is listed here
for some of the stuff they have done in the past) and didn't have the
foggiest clue what they were doing and then get into trouble with
attribute/class collisions later on.

joe


--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net


---O'Reilly Active Directory Third Edition now available---

http://www.joeware.net/win/ad3e.htm

Post by Ib Schrader
Hi
I've tried locating documents or articles that describe how you recover from
a schema change gone bad. I know this is a very serious situation, but I was
hoping there at least was some way of getting back.
I've heard you can keep a DC offline for the upgrade and if everything fails
use that one DC to rebuild the domain, would that work? because I've also
heard that all DC's have to be online for AD to even accept the schema
change.
Is there any way at all to back up the schema and reload it should it become
corrupted in some way?
Thanks for any input.
Ib

[Ib Schrader]

Thanks for all your replies.

I did test the change in a test environment, and the change itself is not
scripted by me, but done by a cisco application that need to extend the
schema to work. So if anyone has that list of clueless vendors where cisco
is supposedly on I'd like to see it :)

The problem is not that I don't trust the application or that I think it
will go wrong. The problem is that my boss asks me what I will do if it goes
wrong, and he doesn't take "it won't" for an answer.

The idea with the lag sit sounds really elegant, I think I'll go with that
one. Or if the schema really does replicate even if one DC is offline I
think I'll do that also. I do realise I'd have to kill all DC's and
completely restore the entire domain from that one DC if the change goes
bad.

Thanks for your input
Ib

[Joe Richards [MVP]]

If you want to see a list of changes:

1. Build a new test forest.
2. Export the schema with ldifde.
3. Run the schema update.
4. Export the schema with ldifde to another file.
5. Get and install ADAM SP1 or R2 (SP1 is a download, R2 is on all R2
servers).
6. Run ADSchemaAnalyzer.exe from the %windir%\adam directory
7. Have it compare the LDIF files and output the changes.

Then you want to look at the new object classes and attributes and
verify the key fields are correct which will help assure uniqueness in
the future. There are quite a few things to check, grab the book in the
signature and read the schema chapters or alternately read the schema
chapters in Inside Active Directory 2nd Edition. Either should be
sufficient to help you figure out if Cisco is dorking anything up with
the schema for this specific product.

Also with Cisco, I have just not been thrilled with any of the apps I
have seen and how they use AD, anytime I have had to investigate one I
have ended up telling the customer to put it in its own forest. But
admittedly it has been a year or two now since I have done one of those
reviews.

--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net


---O'Reilly Active Directory Third Edition now available---

http://www.joeware.net/win/ad3e.htm

Post by Ib Schrader
Thanks for all your replies.
I did test the change in a test environment, and the change itself is not
scripted by me, but done by a cisco application that need to extend the
schema to work. So if anyone has that list of clueless vendors where cisco
is supposedly on I'd like to see it :)
The problem is not that I don't trust the application or that I think it
will go wrong. The problem is that my boss asks me what I will do if it goes
wrong, and he doesn't take "it won't" for an answer.
The idea with the lag sit sounds really elegant, I think I'll go with that
one. Or if the schema really does replicate even if one DC is offline I
think I'll do that also. I do realise I'd have to kill all DC's and
completely restore the entire domain from that one DC if the change goes
bad.
Thanks for your input
Ib

[Paul Bergson [MVP-DS]]

Joe,
I attempted to place a dc offline and do a schema update (Call Manager from
Cisco) and it would not work and I could have sworn that it wanted all dc's
online before it would go forward. So I brought the dc that was offline
back online and everything updated just fine.

This was a good year ago and you are a Braniac so I can't argue with me, but
I was almost positive this was the case.
--
Paul Bergson
MVP - Directory Services
MCT, MCSE, MCSA, Security+, BS CSci
2003, 2000 (Early Achiever), NT

http://www.pbbergs.com

Please no e-mails, any questions should be posted in the NewsGroup
This posting is provided "AS IS" with no warranties, and confers no rights.

Post by Joe Richards [MVP]
1. Build a new test forest.
2. Export the schema with ldifde.
3. Run the schema update.
4. Export the schema with ldifde to another file.
5. Get and install ADAM SP1 or R2 (SP1 is a download, R2 is on all R2
servers).
6. Run ADSchemaAnalyzer.exe from the %windir%\adam directory
7. Have it compare the LDIF files and output the changes.
Then you want to look at the new object classes and attributes and verify
the key fields are correct which will help assure uniqueness in the
future. There are quite a few things to check, grab the book in the
signature and read the schema chapters or alternately read the schema
chapters in Inside Active Directory 2nd Edition. Either should be
sufficient to help you figure out if Cisco is dorking anything up with the
schema for this specific product.
Also with Cisco, I have just not been thrilled with any of the apps I have
seen and how they use AD, anytime I have had to investigate one I have
ended up telling the customer to put it in its own forest. But admittedly
it has been a year or two now since I have done one of those reviews.
--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net
---O'Reilly Active Directory Third Edition now available---
http://www.joeware.net/win/ad3e.htm

Post by Ib Schrader
Thanks for all your replies.
I did test the change in a test environment, and the change itself is not
scripted by me, but done by a cisco application that need to extend the
schema to work. So if anyone has that list of clueless vendors where
cisco is supposedly on I'd like to see it :)
The problem is not that I don't trust the application or that I think it
will go wrong. The problem is that my boss asks me what I will do if it
goes wrong, and he doesn't take "it won't" for an answer.
The idea with the lag sit sounds really elegant, I think I'll go with
that one. Or if the schema really does replicate even if one DC is
offline I think I'll do that also. I do realise I'd have to kill all DC's
and completely restore the entire domain from that one DC if the change
goes bad.
Thanks for your input
Ib

[kj]

As I recall, I had to do a metadata cleanup on a long dead DC before
Exchange 2003 forestprep would run too. I realize this is different, but it
is similar.
--
/kj

Post by Paul Bergson [MVP-DS]
Joe,
I attempted to place a dc offline and do a schema update (Call Manager
from Cisco) and it would not work and I could have sworn that it wanted
all dc's online before it would go forward. So I brought the dc that was
offline back online and everything updated just fine.
This was a good year ago and you are a Braniac so I can't argue with me,
but I was almost positive this was the case.
--
Paul Bergson
MVP - Directory Services
MCT, MCSE, MCSA, Security+, BS CSci
2003, 2000 (Early Achiever), NT
http://www.pbbergs.com
Please no e-mails, any questions should be posted in the NewsGroup
This posting is provided "AS IS" with no warranties, and confers no rights.

Post by Joe Richards [MVP]
1. Build a new test forest.
2. Export the schema with ldifde.
3. Run the schema update.
4. Export the schema with ldifde to another file.
5. Get and install ADAM SP1 or R2 (SP1 is a download, R2 is on all R2
servers).
6. Run ADSchemaAnalyzer.exe from the %windir%\adam directory
7. Have it compare the LDIF files and output the changes.
Then you want to look at the new object classes and attributes and verify
the key fields are correct which will help assure uniqueness in the
future. There are quite a few things to check, grab the book in the
signature and read the schema chapters or alternately read the schema
chapters in Inside Active Directory 2nd Edition. Either should be
sufficient to help you figure out if Cisco is dorking anything up with
the schema for this specific product.
Also with Cisco, I have just not been thrilled with any of the apps I
have seen and how they use AD, anytime I have had to investigate one I
have ended up telling the customer to put it in its own forest. But
admittedly it has been a year or two now since I have done one of those
reviews.
--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net
---O'Reilly Active Directory Third Edition now available---
http://www.joeware.net/win/ad3e.htm

Post by Ib Schrader
Thanks for all your replies.
I did test the change in a test environment, and the change itself is
not scripted by me, but done by a cisco application that need to extend
the schema to work. So if anyone has that list of clueless vendors where
cisco is supposedly on I'd like to see it :)
The problem is not that I don't trust the application or that I think it
will go wrong. The problem is that my boss asks me what I will do if it
goes wrong, and he doesn't take "it won't" for an answer.
The idea with the lag sit sounds really elegant, I think I'll go with
that one. Or if the schema really does replicate even if one DC is
offline I think I'll do that also. I do realise I'd have to kill all
DC's and completely restore the entire domain from that one DC if the
change goes bad.
Thanks for your input
Ib

[Joe Richards [MVP]]

It is the application checking... A schema mod is a simple LDAP update
just like any other update. You simply need to have the Schema FSMO
machine up and running. When you go to make the update to any machine,
you will get referred to the Schema FSMO and bam the change will be made.

The main differences are that the updates have to be sent to a specific
DC (via directly or through referral as mentioned) and it isn't easy to
delete the added class and attribute objects.

joe

--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net


---O'Reilly Active Directory Third Edition now available---

http://www.joeware.net/win/ad3e.htm

Post by Paul Bergson [MVP-DS]
Joe,
I attempted to place a dc offline and do a schema update (Call Manager from
Cisco) and it would not work and I could have sworn that it wanted all dc's
online before it would go forward. So I brought the dc that was offline
back online and everything updated just fine.
This was a good year ago and you are a Braniac so I can't argue with me, but
I was almost positive this was the case.

[Ib Schrader]

So there is no restriction from Microsoft's side that prevents a schema
modification to be performed if any DC's are offline, so if an application
refuses to do it due to an offline DC it's the design of the
application..and not the underlying AD that refuses to do it?

If that is the case I think I might need to make a lag site only..if the
cisco application (Cisco Unity)simply won't update the schema with any DC's
offline.

Does anyone know if schema modifications replicate alongside other AD
updates like normal user creation and things like that?..I mean if I move a
DC to a test site and configure a really slow update time to that site the
schema wont update earlier due to some priority replication or
schema-specific replication or something??

Thanks for your answers
Ib

[Joe Richards [MVP]]

Correct, AD enforces no DC availability for schema mods.

When a DC has an updated schema, its partners know it and nothing else
can be replicated FROM that DC to any other DC until the schema has
replicated. It will not impact replication between any other DC's and
their replication partners. So for example, say you have

DC1
DC2
DC3
DC4

DC1 replicates with DC2 which replicates with DC3 and DC4.

If you update the schema on DC1 and then add a user to DC1, the user add
on DC1 will not replicate to DC2 until after the schema update has
replicated.

However, DC2,3,4 can replicate normal changes just fine between each
other. Once DC2 gets it, its partners will not be able to pull from it
until they pull the schema updates.

--
Joe Richards Microsoft MVP Windows Server Directory Services
Author of O'Reilly Active Directory Third Edition
www.joeware.net


---O'Reilly Active Directory Third Edition now available---

http://www.joeware.net/win/ad3e.htm

Post by Ib Schrader
So there is no restriction from Microsoft's side that prevents a schema
modification to be performed if any DC's are offline, so if an application
refuses to do it due to an offline DC it's the design of the
application..and not the underlying AD that refuses to do it?
If that is the case I think I might need to make a lag site only..if the
cisco application (Cisco Unity)simply won't update the schema with any DC's
offline.
Does anyone know if schema modifications replicate alongside other AD
updates like normal user creation and things like that?..I mean if I move a
DC to a test site and configure a really slow update time to that site the
schema wont update earlier due to some priority replication or
schema-specific replication or something??
Thanks for your answers
Ib