Active Directory Hell

Active Directory Hell

(from RussellMUCK 07/30/2003)

Calin AARRRGS, and throws Windows 2000 Active Directory Domains out the freakin' window!!!

Calin turns off his 'damage control' sirens for the moment.

Akili rumbles, "Yikes!"

Calin says, "Ok, so I have a story:"

Akili settles in to listen.

Calin says, "We were using two Windows NT domains (WORKGROUP and SNFLWR). Yes, WORKGROUP was named thus as a tribute to lazy admins."

Akili snickers. "Okay."

Akili thinks he remembers that, actually.

Calin says, "We had one domain controller for each domain, and SEATTLE, the WORKGROUP domain controller, was running Exchange 5.5 for email."

Akili rumbles, "Okay."

Calin says, "Exchange on Seattle handled email for both domains. The domains trusted each other, so it was no trouble to tell Exchange that the owner account for a mailbox is in the other domain."

Calin says, "Now, in the last few months, spam has become a SERIOUS problem here."

Akili rumbles, "Yeah, there's some of that at my office, too. Not everyone, but certain accounts are getting hammered."

Calin says, "So I started looking around for a spam filter program to put in."

Calin says, "Almost everything I found worked only for Exchange 2000, not Exchange 5.5."

Calin says, "Eventually we bought one that needed to run on windows 2000, but could forward all email to exchange 5.5."

Calin says, "So that's what we did."

Akili rumbles, "Okay."

Calin says, "So anyway, it was running as our smtp gateway, and it filtered away."

Akili rumbles, "Sounds good."

Calin says, "We quickly realized though, that it was blocking many valid emails."

Akili laughs. "Oops."

Calin says, "So I turned down it's abilities, and turned them down, until it was hardly blocking anything."

Calin says, "So then it only blocked based on the sender's email address, and we started a list."

Cal says, "It wasn't the filtering."

Calin says, "We still blocked hundreds of emails, but so many kept getting through."

Calin says, "With a bit more exploring, I found a feature called 'auto white list.'"

Akili rumbles, "Yeah. List filtering is only useful against valid advertisers, and those aren't the problem anyway."

Calin says, "This was our savior."

Akili rumbles, "Auto white list?"

Calin says, "You see, any time we SENT an email, the recipient got added to the white list."

Calin says, "Which means they'd never be blocked."

Akili rumbles, "Ahh. Interesting."

Calin says, "So I figured I'd run it that way for a while, only blocking by address but building a white list... and after the white list got to be pretty big, I'd turn on the other features again."

Calin says, "I ran it for a week or two, and took a look at the white list. It sure was getting big."

Calin says, "To my horror though, I saw a whole bunch of obviously spammy addresses in there."

Calin says, "Well what the heck??"

Akili raises a brow.

Akili rumbles, "Ooh. I hope the spams weren't automatically trying to send a delivery notification."

Calin says, "A bit of investigation turned up the fact that any time the server sent an NDR (Non Delivery Reply) it was adding the recipient to the whitelist."

Akili rumbles, "Ack!"

Calin says, "That's what _I_ thought."

Calin says, "So I emailed the maker of the filter, and asked them about it."

Calin says, "They said it was a known bug, and to wait for the next build."

Akili rolls his eyes.

Calin says, "I got the next build a few days later."

Calin says, "Then I cleared out the whitelist and started again."

Calin says, "A week or so later I checked it, and found that once again, it was filled with spammy email addresses."

Calin says, "Again, what the heck?"

Calin says, "WELL."

Calin says, "Here's the deal, sparky."

Akili grins. "This should be good. In a really awful sort of way."

Calin says, "Because it was only a gateway for Exchange 5.5, and not integrated with it... it couldn't tell the difference between an NDR generated by Exchange 5.5 and a valid email. The bug was apparently coincidental, and applied only to NDRs generated by the machine with the filter on it."

Calin says, "So now we're left with a (mostly) useless email filter."

Akili rumbles, "Wonderful."

Akili rumbles, "Refund!"

Calin says, "Time went by, and I dutifully added over 3000 emails to the junk senders list."

Akili rumbles, "To little avail, I'd imagine."

Calin says, "Eventually my boss asked me what we could do about it."

Calin says, "Actually, it does block a good amount."

Akili rumbles, "Really? That's surprising."

Calin says, "Yeah. Lots more get through though."

Calin says, "Anyway, what I told him was this: In order to get the best use of the filter, we need to use Exchange 2000."

Akili rumbles, "Most senders I've seen that use the same address are generally pretty good about acknowledging unsubscription requests."

Akili rumbles, "Of course."

Calin says, "He asked what it would take to get that running, and I thought about it for a while..."

Calin says, "We had a server available. This gave me some options."

Calin says, "Firstly, Exchange 2000 NEEDS to run in a windows 2000 active directory domain, because it integrates with it."

Akili rumbles, "Right. That's why we haven't upgraded at work either."

Akili rumbles, "We're working on it, though."

Calin says, "Right. So that means upgrading at least one of my domains to windows 2000 active directory."

Akili rumbles, "Let the nightmare commence."

Calin says, "So I figured I'd install Windows 2000 Server on the machine I had available, and have it take over the Workgroup domain, as well as handle many of the functions of seattle, the current workgroup controller."

Akili yodas, "Not ready are you. Of the Dark Side it is."

Calin says, "So, I install windows 2000 server. So far so good."

Calin says, "The mouse doesn't work, and I have to install a 3rd party driver for it, but no big deal."

Calin says, "I run DCPROMO, to promote it to a domain controller."

Calin says, "It tells me that it can't join WORKGROUP, because it cannot contact the current Active Directory controller."

Akili rumbles, "Uh-oh."

Calin says, "It tells me I cannot create a new domain with the NT compatible name WORKGROUP because it already exists."

Calin says, "So I backtrack. I install NT server on the machine."

Calin says, "During install I need to find and provide a SCSI driver, a network driver, and a mouse driver, but I finally get it installed and set up as a Backup Domain Controller for WORKGROUP. "

Akili rumbles, "You mentioned that you were having to rebuild an NT server, yeah."

Calin says, "I use the Server Manager to promote the new machine to the Primary Domain Controller. It works flawlessly, demoting Seattle to be the Backup Domain Controller."

Calin says, "And I'm thinking, 'Wow, this actually works!'"

Calin says, "Who'd have thunk it."

Calin says, "Then, I install patch 6 on the new machine."

Cal says, "clunk."

Akili rumbles, "Foom?"

Calin says, "Nah, still going well here."

Calin says, "Then, I upgraded it to windows 2000."

Akili rumbles, "Falling flaming bits from the sky?"

Calin says, "As soon as the upgrade was complete, it automatically ran DCPROMO, and created a new domain called WORKGROUP (for NT) and annams.com (for win2k)."

Calin says, "Seattle continued to work just fine as a Backup Domain Controller. Things were looking well."

Calin says, "Again, I'm thinking, 'Wow, this works!'"

Akili rumbles, "And is that when you accidently knocked the gearshift from 5th to 1st and flung engine parts across the freeway?"

Calin says, "I installed Exchange 2000 on the new server, and joined it to the existing Exchange 5.5 organization."

Calin says, "Now I could view all the mailboxes on Exchange 5.5, from the Exchange 2000 console."

Calin says, "Looking good..."

Calin says, "I dinked with it for a bit, and finally found the option to replicate data from the 5.5 server to the 2000 server."

Calin says, "I ran it..."

Calin says, "And found that it did horrible, horrible things."

Akili rumbles, "System go down the hole?"

Calin says, "It copied all the mailboxes from the old system..."

Calin says, "And created user accounts for all the ones that didn't have one in WORKGROUP."

Akili rumbles, "Uh."

Calin says, "It seems that in Exchange 2000, you cannot assign ownership of a mailbox."

Cal says, "you did something they didn't think of."

Calin says, "It's built into the active directory account of the owner."

Calin says, "Yeah, that's the impression I get, Cal."

Akili rumbles, "Well. That's messed up."

Calin says, "It seems it never OCCURED to anyone that you'd use ONE exchange server for more than one domain."

Akili rumbles, "Sounds like Microsoft."

Calin says, "So anyway, I killed the win2000 server, promoted Seattle back to PDC, and deleted all the erroneously created accounts."

Calin says, "I started over."

Calin says, "I installed NT, took control of the domain, installed 2k."

Calin says, "I figured I needed SNFLWR to be a win2k domain too, so it's users would be in the Active Directory."

Calin says, "I didn't have a machine available, so I used VMware to make one."

Akili rumbles, "Okay."

Calin says, "On the annams.com active directory controller, I started a vmware, installed NT, took control of SNFLWR, and installed windows 2000."

Akili shakes his head. "Just from what you've gone through already... I'm not looking forward to this project at my office."

Calin says, "A pretty ballsy move, but I didn't see any reason it shouldn't work."

Akili chuckles. "I'd agree."

Calin says, "It did work. I created the snflwr.com domain as a peer to annams.com."

Calin says, "It all seemed to be working just fine, but I noticed some odd things in the Event Logs."

Akili raises a brow.

- Error: You're doing something we don't like. You will be paying for it shortly.

Calin says, "It seemed the snflwr.com controller couldn't communicate with the Global Catalog (whatever that is) on the annams.com controller."

Calin says, "Users started coming to find me."

Akili rumbles, "That's never good."

Akili rumbles, "They almost never find the sysadmin to give him good news."

Calin says, "It seems that the domain controllers couldn't talk to each other properly, so the trust between the domains was broken."

Akili rumbles, "That's like a cop pulling you over to tell you how good a driver you are."

Calin says, "So users in one domain could no longer access resources in the other."

Akili rumbles, "That's bad."

Calin says, "Yup."

Calin says, "With some trouble, I was able to re-establish the trusts."

Calin says, "But there were still problems. People stopped coming to find me though, so it was ok."

Akili grins.

Calin says, "I checked things out, and found that Exchange (when installed on the annams.com controller) could do nothing with the accounts on the snflwr.com controller."

Calin says, "This kind of made sense to me. It was because exchange had extended the directory to support it on one, but not the other."

Calin says, "So I went about installing it on the snflwr.com controller."

Calin says, "It wouldn't install."

Calin says, "It said it didn't have access to modify the Schema."

Calin says, "So I couldn't install it."

Akili shakes his head.

Calin says, "I figured perhaps there was something odd about the VM that was keeping it from working."

Calin says, "So I took the win2k server that was currently running the spam filter (as a member server) and promoted it to a domain controller, adding it to the snflwr.com domain."

Calin says, "It could communicate just fine with the VM server."

Calin says, "Then I took the VM server offline, and tried to use just the new server as the snflwr.com controller."

Calin says, "But it seems that the new server synchronized with the VM server, and inherited some 'out of sync' stuff. So it simply couldn't talk to the annams.com controller."

Calin says, "Now I was in a quandary."

Calin says, "This machine was neither working, nor could I take it offline (as it was still our SMTP gateway)."

Akili rumbles, "Right."

Calin says, "I tried to demote it, but it said 'Missing DSA' or something."

Calin says, "So it was a server, and it was staying that way."

Calin says, "Eventually I shut down all the services relating to netbios and netlogon, and left it running."

Calin says, "So as far as the rest of the windows world was concerned, it wasn't there."

Calin says, "I promoted the old NT server back to being the PDC."

Calin says, "And I recreated the trust, which had gone bad again."

Akili chuckles, recalling the server that wasn't there at TRS.

Calin says, "I cobbled together some old parts, and got myself a new machine."

Calin says, "I went through it all again, putting on NT, taking control of SNFLWR, and installing win2k."

Akili shakes his head. "Man."

Calin says, "The trusts immediately went bad."

Calin says, "I recreated them. "

Calin says, "Users started to come to me."

Calin says, "It seems that when I recreated the trusts this time, it was only a trust the win2k servers could use."

Calin says, "The two NT backup controllers could not talk to each other."

Calin says, "And for some reason didn't want to trust the win2k controllers of the other domain."

Calin says, "Thus any services that still lived on the NT servers became 'off limits' to users of the opposite domain."

Calin says, "This included email."

Calin says, "I struggled to fix it, and users kept coming to me."

Akili covers his face with a hand, and continues shaking his head. "Jeez."

Calin says, "Finally, I pulled the plug on both win2k servers, and promoted both NT servers to primary again."

Calin says, "I re-established the trust."

Calin says, "Things started working again, with one exception:"

Akili rumbles, "Email?"

Calin says, "Any windows 2000 machine that had been rebooted since the win2k servers came up had dutifully changed their domain membership from WORKGROUP to annams.com, or from SNFLWR to snflwr.com."

Calin says, "They could no longer log in."

Akili rumbles, "Arg."

Calin says, "The computer accounts in the domain were now orphaned, and had to be deleted manually."

Calin says, "The computers themselves had to be removed from the domain and readded."

Calin says, "And that brings us to now. Two windows NT domain controllers, Zero windows 2k controllers...."

Calin says, "And undoubtely a few computers just waiting for tomorrow morning before deciding that their machine accounts are bad."

Akili rumbles, "What a mess."

Calin leans out the window, and watches Active Directory smash to a million bits on the ground.

Calin says, "At a guess, I would say that win2k domains can work with NT compatibility on."

- The pieces liquify, start rolling together, and reform, shaking a furious fist at Calin.

Calin says, "However, I think that if you have trust relationships between win2k domains, one of them MUST be in native mode."

Calin says, "Which means no NT servers in the domain."

Calin says, "Which means it's not going to work here."

Calin says, "The other option would be to start a new win2k domain, and then upgrade the existing domains as children of the new native domain."

Calin says, "That might work."

Calin says, "however, I don't htink I'm going to do that just now."

Akili rumbles, "I don't blame you."

Calin shakes his head.

Calin says, "I'm tired after just writing it all, much less doing it."

Akili rumbles, "And this is why you have no spare time."

Calin says, "This is EXACTLY why I have no spare time."

Calin waves his hand in a lazy circular gesture. "That's it, I'm going back to bed."