ADAM - The Server is not operational

Discussion:

(too old to reply)

2006-02-05 20:34:49 UTC

Interested in some questions around the limits or ADAM.

I have created a simple test console app that fires off multiple threads.
With each thread a single user is created and added to the readers group.
The thread then terminates. This is much like a typical registration on a
web page. I have noticed that on two of my machines (completely different
specs), the limit is around 300 concurrent connections to the ADAM server
before i get "The server is not operational" on a bind to the directory
store.

Is this conincidence or is there something around the 300 mark i should be
aware of? 300 is not a bad number - especially on two machines that aren't
server machines, but it is how close they are to 300 (and again one is twice
as fast as the other and double the memory) that makes me think.

It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting factors on
ADAM connections are? In other words, what causes the "Server is not
operational" error to happen, even though it does actually work for those
that made a successful bind? My memory seems ok, my cpu isn't hit too hard.

I'd love to get into the details of this (even if it brings up adsi and so
on). It may be useful for me this week!

Regards,
steven :: http://stevenR2.com

Dmitri Gavrilov [MSFT]

2006-02-06 09:45:32 UTC

Permalink

Looks like you are running out of ephemeral ports on the client. Look at
this KB that describes a similar problem (sorry, could not find anything
better than that):
http://support.microsoft.com/?id=836429

The resolution is to reuse ldap connections.
--
Dmitri Gavrilov
SDE, DS Admin eXperience

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple threads.
With each thread a single user is created and added to the readers group.
The thread then terminates. This is much like a typical registration on a
web page. I have noticed that on two of my machines (completely different
specs), the limit is around 300 concurrent connections to the ADAM server
before i get "The server is not operational" on a bind to the directory
store.
Is this conincidence or is there something around the 300 mark i should be
aware of? 300 is not a bad number - especially on two machines that aren't
server machines, but it is how close they are to 300 (and again one is
twice as fast as the other and double the memory) that makes me think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting factors
on ADAM connections are? In other words, what causes the "Server is not
operational" error to happen, even though it does actually work for those
that made a successful bind? My memory seems ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi and so
on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

2006-02-06 10:28:23 UTC

Permalink

Thanks Dmitri - so long as i can understand what is going on, anything is a
great help!

In a scenario whereby all my clients are web clients, it is a good idea to
create your own LDAP connection pool?
I know such a thing is handled by ODBC drivers and/or managed code for
connections to databases, but in the case of ldap connections to ADAM, is
this something that should be written ourselves?

Best Regards,
steven :: http://stevenR2.com

Post by Dmitri Gavrilov [MSFT]
Looks like you are running out of ephemeral ports on the client. Look at
this KB that describes a similar problem (sorry, could not find anything
http://support.microsoft.com/?id=836429
The resolution is to reuse ldap connections.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple threads.
With each thread a single user is created and added to the readers group.
The thread then terminates. This is much like a typical registration on a
web page. I have noticed that on two of my machines (completely different
specs), the limit is around 300 concurrent connections to the ADAM server
before i get "The server is not operational" on a bind to the directory
store.
Is this conincidence or is there something around the 300 mark i should
be aware of? 300 is not a bad number - especially on two machines that
aren't server machines, but it is how close they are to 300 (and again
one is twice as fast as the other and double the memory) that makes me
think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting factors
on ADAM connections are? In other words, what causes the "Server is not
operational" error to happen, even though it does actually work for those
that made a successful bind? My memory seems ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi and
so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

2006-02-06 10:40:44 UTC

Permalink

Also, i had a look at what reflector gives me for the dispose method of the
DirectoryEntry.
The close() calls unbind().

Unless there is some magic happening whereby connections are reused under
the covers, then i'd suspect that each timem you call dispose you have to do
a full rebind for each and every client. Would it be safe to not close or
dispose these objects in this way (either in "using" or by calling dispose)
and keep them open and reuse the connection.

The idea is that some connection factory could manage this for me, but i
know in the world of ADO connections this is not a good thing to do as it's
all managed and optimized for you. I wonder whether it is the same for adam.

Regards,
steven :: http://stevenR2.com

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple threads.
With each thread a single user is created and added to the readers group.
The thread then terminates. This is much like a typical registration on a
web page. I have noticed that on two of my machines (completely different
specs), the limit is around 300 concurrent connections to the ADAM server
before i get "The server is not operational" on a bind to the directory
store.
Is this conincidence or is there something around the 300 mark i should
be aware of? 300 is not a bad number - especially on two machines that
aren't server machines, but it is how close they are to 300 (and again
one is twice as fast as the other and double the memory) that makes me
think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting factors
on ADAM connections are? In other words, what causes the "Server is not
operational" error to happen, even though it does actually work for those
that made a successful bind? My memory seems ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi and
so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

Dmitri Gavrilov [MSFT]

2006-02-06 17:05:02 UTC

Permalink

You should chat with Joe Kaplan on ADSI newsgroup (though he is here too).
ADSI does do connection pooling for you, and S.DS does use ADSI. However,
ADSI only holds the pool as long as you have outstanding COM references to
its objects. Which means, if you call dispose, you lose it.

Now, there's another complication. AFAIK, ADSI pools connections based on
credentials. So, if you are binding using different creds, then you are not
making use of the pooling. Joe can probably provide you with more info and
workarounds.
--
Dmitri Gavrilov
SDE, DS Admin eXperience

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Also, i had a look at what reflector gives me for the dispose method of
the DirectoryEntry.
The close() calls unbind().
Unless there is some magic happening whereby connections are reused under
the covers, then i'd suspect that each timem you call dispose you have to
do a full rebind for each and every client. Would it be safe to not close
or dispose these objects in this way (either in "using" or by calling
dispose) and keep them open and reuse the connection.
The idea is that some connection factory could manage this for me, but i
know in the world of ADO connections this is not a good thing to do as
it's all managed and optimized for you. I wonder whether it is the same
for adam.
Regards,
steven :: http://stevenR2.com

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a typical
registration on a web page. I have noticed that on two of my machines
(completely different specs), the limit is around 300 concurrent
connections to the ADAM server before i get "The server is not
operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i should
be aware of? 300 is not a bad number - especially on two machines that
aren't server machines, but it is how close they are to 300 (and again
one is twice as fast as the other and double the memory) that makes me
think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the "Server
is not operational" error to happen, even though it does actually work
for those that made a successful bind? My memory seems ok, my cpu isn't
hit too hard.
I'd love to get into the details of this (even if it brings up adsi and
so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

2006-02-06 17:42:50 UTC

Permalink

Dmitri - speaking with Joe.
Yes, I see what you mean about the credentials - due to the way
ADsOpenObject works underneath.

Thanks for your post!

steven :: http://stevenR2.com

Post by Dmitri Gavrilov [MSFT]
You should chat with Joe Kaplan on ADSI newsgroup (though he is here too).
ADSI does do connection pooling for you, and S.DS does use ADSI. However,
ADSI only holds the pool as long as you have outstanding COM references to
its objects. Which means, if you call dispose, you lose it.
Now, there's another complication. AFAIK, ADSI pools connections based on
credentials. So, if you are binding using different creds, then you are
not making use of the pooling. Joe can probably provide you with more info
and workarounds.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a typical
registration on a web page. I have noticed that on two of my machines
(completely different specs), the limit is around 300 concurrent
connections to the ADAM server before i get "The server is not
operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i should
be aware of? 300 is not a bad number - especially on two machines that
aren't server machines, but it is how close they are to 300 (and again
one is twice as fast as the other and double the memory) that makes me
think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the
"Server is not operational" error to happen, even though it does
actually work for those that made a successful bind? My memory seems
ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi and
so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

Joe Kaplan (MVP - ADSI)

2006-02-06 17:44:22 UTC

Permalink

We have a parallel thread going on the group where we are discussing this
very thing. :)

If different credentials are used under high load with ADSI, it just won't
scale, as the model uses a different LDAP connection for each set of
credentials and we'll evetually run out of wildcard ports.

However, if we use one set of credentials (a service account, for example),
then ADSI should scale. The trick here is to keep one object that has
opened an LDAP connection alive for the duration of the app. There are a
ton of different ways to do this in an ASP.NET app (static variables,
ASP.NET cache, custom cache, etc.).

Joe K.

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a typical
registration on a web page. I have noticed that on two of my machines
(completely different specs), the limit is around 300 concurrent
connections to the ADAM server before i get "The server is not
operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i should
be aware of? 300 is not a bad number - especially on two machines that
aren't server machines, but it is how close they are to 300 (and again
one is twice as fast as the other and double the memory) that makes me
think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the
"Server is not operational" error to happen, even though it does
actually work for those that made a successful bind? My memory seems
ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi and
so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

2006-02-06 18:01:50 UTC

Permalink

Was my fault as i changed to the adsi group :S
I understand this - this is how i understand it to work from the docs.

Server.Transfer("microsoft.public.adsi.general");

thanks guys.

Post by Joe Kaplan (MVP - ADSI)
We have a parallel thread going on the group where we are discussing this
very thing. :)
If different credentials are used under high load with ADSI, it just won't
scale, as the model uses a different LDAP connection for each set of
credentials and we'll evetually run out of wildcard ports.
However, if we use one set of credentials (a service account, for
example), then ADSI should scale. The trick here is to keep one object
that has opened an LDAP connection alive for the duration of the app.
There are a ton of different ways to do this in an ASP.NET app (static
variables, ASP.NET cache, custom cache, etc.).
Joe K.

Post by Dmitri Gavrilov [MSFT]
You should chat with Joe Kaplan on ADSI newsgroup (though he is here
too). ADSI does do connection pooling for you, and S.DS does use ADSI.
However, ADSI only holds the pool as long as you have outstanding COM
references to its objects. Which means, if you call dispose, you lose it.
Now, there's another complication. AFAIK, ADSI pools connections based on
credentials. So, if you are binding using different creds, then you are
not making use of the pooling. Joe can probably provide you with more
info and workarounds.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Also, i had a look at what reflector gives me for the dispose method of
the DirectoryEntry.
The close() calls unbind().
Unless there is some magic happening whereby connections are reused
under the covers, then i'd suspect that each timem you call dispose you
have to do a full rebind for each and every client. Would it be safe to
not close or dispose these objects in this way (either in "using" or by
calling dispose) and keep them open and reuse the connection.
The idea is that some connection factory could manage this for me, but i
know in the world of ADO connections this is not a good thing to do as
it's all managed and optimized for you. I wonder whether it is the same
for adam.
Regards,
steven :: http://stevenR2.com

Post by Dmitri Gavrilov [MSFT]
Looks like you are running out of ephemeral ports on the client. Look
at this KB that describes a similar problem (sorry, could not find
http://support.microsoft.com/?id=836429
The resolution is to reuse ldap connections.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a typical
registration on a web page. I have noticed that on two of my machines
(completely different specs), the limit is around 300 concurrent
connections to the ADAM server before i get "The server is not
operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i
should be aware of? 300 is not a bad number - especially on two
machines that aren't server machines, but it is how close they are to
300 (and again one is twice as fast as the other and double the
memory) that makes me think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the
"Server is not operational" error to happen, even though it does
actually work for those that made a successful bind? My memory seems
ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi
and so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

Eric Fleischman [MSFT]

2006-02-07 16:43:28 UTC

Permalink

You can also increase the # of ephemeral ports. If memory serves me
correctly, we only use up to port 5000 by default, however the spec allows
for 16 bit port numbers. So you could crank it up to say 40000 or something,
assuming your server can scale to this.
Further, if needed, you can turn down the amt of time that ports sit in
time_wait. HOWEVER, in so doing you risk packets still in flight hitting the
port after it has been recycled. So, you'd want to carefully understand your
network and the LDAP clients that will be hitting ADAM before you travel
down this road. Hopefully, increasing the # of ports available is enough for
you.

~Eric
--
Eric Fleischman [MSFT]
Microsoft Windows Server Division
This post is provided "AS IS" with no warranties, and confers no rights

Post by Dmitri Gavrilov [MSFT]
You should chat with Joe Kaplan on ADSI newsgroup (though he is here
too). ADSI does do connection pooling for you, and S.DS does use ADSI.
However, ADSI only holds the pool as long as you have outstanding COM
references to its objects. Which means, if you call dispose, you lose it.
Now, there's another complication. AFAIK, ADSI pools connections based on
credentials. So, if you are binding using different creds, then you are
not making use of the pooling. Joe can probably provide you with more
info and workarounds.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Also, i had a look at what reflector gives me for the dispose method of
the DirectoryEntry.
The close() calls unbind().
Unless there is some magic happening whereby connections are reused
under the covers, then i'd suspect that each timem you call dispose you
have to do a full rebind for each and every client. Would it be safe to
not close or dispose these objects in this way (either in "using" or by
calling dispose) and keep them open and reuse the connection.
The idea is that some connection factory could manage this for me, but i
know in the world of ADO connections this is not a good thing to do as
it's all managed and optimized for you. I wonder whether it is the same
for adam.
Regards,
steven :: http://stevenR2.com

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a typical
registration on a web page. I have noticed that on two of my machines
(completely different specs), the limit is around 300 concurrent
connections to the ADAM server before i get "The server is not
operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i
should be aware of? 300 is not a bad number - especially on two
machines that aren't server machines, but it is how close they are to
300 (and again one is twice as fast as the other and double the
memory) that makes me think.
It is interesting to me that this limitation is on the bind and not on
saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the
"Server is not operational" error to happen, even though it does
actually work for those that made a successful bind? My memory seems
ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi
and so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com

Joe Kaplan (MVP - ADSI)

2006-02-07 17:33:31 UTC

Permalink

This is good info. My hope is that if he gets his connection caching in
ADSI/SDS working as it is supposed to, he'll just have 1 LDAP connection
going under the hood and won't need to bother with this. I haven't heard
follow up yet though.

Joe K.

Post by Eric Fleischman [MSFT]
You can also increase the # of ephemeral ports. If memory serves me
correctly, we only use up to port 5000 by default, however the spec allows
for 16 bit port numbers. So you could crank it up to say 40000 or
something, assuming your server can scale to this.
Further, if needed, you can turn down the amt of time that ports sit in
time_wait. HOWEVER, in so doing you risk packets still in flight hitting
the port after it has been recycled. So, you'd want to carefully
understand your network and the LDAP clients that will be hitting ADAM
before you travel down this road. Hopefully, increasing the # of ports
available is enough for you.
~Eric

Eric Fleischman [MSFT]

2006-02-07 19:57:04 UTC

Permalink

That is definitely a better solution if you can make it work. Connection
recycling saves you the time and perf cost of rebuilding connections over
and over. But if it is required, this is doable.
--
Eric Fleischman [MSFT]
Microsoft Windows Server Division
This post is provided "AS IS" with no warranties, and confers no rights

Post by Joe Kaplan (MVP - ADSI)
This is good info. My hope is that if he gets his connection caching in
ADSI/SDS working as it is supposed to, he'll just have 1 LDAP connection
going under the hood and won't need to bother with this. I haven't heard
follow up yet though.
Joe K.

Post by Eric Fleischman [MSFT]
You can also increase the # of ephemeral ports. If memory serves me
correctly, we only use up to port 5000 by default, however the spec
allows for 16 bit port numbers. So you could crank it up to say 40000 or
something, assuming your server can scale to this.
Further, if needed, you can turn down the amt of time that ports sit in
time_wait. HOWEVER, in so doing you risk packets still in flight hitting
the port after it has been recycled. So, you'd want to carefully
understand your network and the LDAP clients that will be hitting ADAM
before you travel down this road. Hopefully, increasing the # of ports
available is enough for you.
~Eric

2006-02-07 20:28:56 UTC

Permalink

Thanks Eric and Joe - I had read up on quite a bit of ephemeral ports and
TIME_WAIT.
Found a good blog here
http://www.port80software.com/200ok/archive/2004/12/07/205.aspx

Unfortunately i won't find out the exact circumstances until Friday when i
go on site (i just have some code snippets and some issues).
My suspicision is that they don't use connection caching and they also use
RefreshCache() rather than specifying the attributes to refresh which must
be a hit!

There will be one ldap connection for some ops. However, each user (created
as a new user in ADAM) will have to then be able to log on as themselves and
set their user properties.
This will likely use SimpleBind and the upn and pwd that was previously set.
This i guess runs as a new connection.

I'd have to think about why they don't use NLB rather than a single server
with an increased number of ephemeral ports or minimising tcp time wait.

steven :: http://stevenR2.com

Post by Joe Kaplan (MVP - ADSI)
We have a parallel thread going on the group where we are discussing this
very thing. :)
If different credentials are used under high load with ADSI, it just
won't scale, as the model uses a different LDAP connection for each set
of credentials and we'll evetually run out of wildcard ports.
However, if we use one set of credentials (a service account, for
example), then ADSI should scale. The trick here is to keep one object
that has opened an LDAP connection alive for the duration of the app.
There are a ton of different ways to do this in an ASP.NET app (static
variables, ASP.NET cache, custom cache, etc.).
Joe K.

Post by Dmitri Gavrilov [MSFT]
You should chat with Joe Kaplan on ADSI newsgroup (though he is here
too). ADSI does do connection pooling for you, and S.DS does use ADSI.
However, ADSI only holds the pool as long as you have outstanding COM
references to its objects. Which means, if you call dispose, you lose it.
Now, there's another complication. AFAIK, ADSI pools connections based
on credentials. So, if you are binding using different creds, then you
are not making use of the pooling. Joe can probably provide you with
more info and workarounds.
--
Dmitri Gavrilov
SDE, DS Admin eXperience
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by sl
Also, i had a look at what reflector gives me for the dispose method of
the DirectoryEntry.
The close() calls unbind().
Unless there is some magic happening whereby connections are reused
under the covers, then i'd suspect that each timem you call dispose you
have to do a full rebind for each and every client. Would it be safe to
not close or dispose these objects in this way (either in "using" or by
calling dispose) and keep them open and reuse the connection.
The idea is that some connection factory could manage this for me, but
i know in the world of ADO connections this is not a good thing to do
as it's all managed and optimized for you. I wonder whether it is the
same for adam.
Regards,
steven :: http://stevenR2.com

Post by sl
Interested in some questions around the limits or ADAM.
I have created a simple test console app that fires off multiple
threads. With each thread a single user is created and added to the
readers group. The thread then terminates. This is much like a
typical registration on a web page. I have noticed that on two of my
machines (completely different specs), the limit is around 300
concurrent connections to the ADAM server before i get "The server is
not operational" on a bind to the directory store.
Is this conincidence or is there something around the 300 mark i
should be aware of? 300 is not a bad number - especially on two
machines that aren't server machines, but it is how close they are to
300 (and again one is twice as fast as the other and double the
memory) that makes me think.
It is interesting to me that this limitation is on the bind and not
on saving data and so on - does anyone know what causes the limiting
factors on ADAM connections are? In other words, what causes the
"Server is not operational" error to happen, even though it does
actually work for those that made a successful bind? My memory seems
ok, my cpu isn't hit too hard.
I'd love to get into the details of this (even if it brings up adsi
and so on). It may be useful for me this week!
Regards,
steven :: http://stevenR2.com