Developer Forum

We have 4x150 users licence, set to unlimited bandwith.

We get these messages in event viewer:
Connection rejected by server. Reason : [ License.Limit.Exceeded ] :
(_defaultRoot_, _defaultVHost_) : Max connections allowed exceeds license
limit. Rejecting connection to : our_app_name/.

It appends a lot since the last few days, even before last update 2.0.3

At the same time, when we check the administration console, the number in the
_defaultVhost is much lowe than 600, the maximum number of user that is allowed.

Licences are all OK in the administration console.

We have to restart the Flash server service to correct the problem.

Please help...

Hello,

We have always experienced that same issue with fms. I would be eager to share
thoughts on solutions with you. We operate 2 licences (1000-40pro) on a box and
experience a fmscore.exe hang which results in the license max exceeded as
people try to connect to the stalled core. So far we have only been able to
find a way to minimize the impact of such an event by configuring fms to spawn
a core for every application instance.

I'll email you through your flashcomguru post so that you might benefit from
each other's experience with this problem.

If I find a solution in the meantime I'll post it here for you as well as on
your flashcomguru posting.

Regards,

Dean

We also operate two 1000-40 stacked licenses and get the same
[License.Limit.Exceeded] Warnings in our Event Log. We have rolled back to
version 2.0.1 because this issue started in 2.0.2 and continued into 2.0.3. If
anyone knows of a workaround or fix we would appreciate the input as we would
like to run the most recent version of the software but cannot due to this bug.
Does Macromedia monitor these forums?

we've experienced this problem 2 days ago either on version 2.0.1 r27

This issue can have different reasons - it's not necessarily a bug. Please note
that FMS counts connections, not users, so if one client creates more than 1
connections, it's expected behaviour that you run into the license limit.

To make sure that this is not the case, please describe your application in
more detail (what is the application doing, what happens when a user connects).
Without this information it's tough to say what's going wrong.

Hello mm_Patrick,

Thank you for your interest in this problem.

We restrict multiple access to the same application instance on our service.

We operate a chat using fms. We experience, from time to time, an application
instance failure which results in the entire room participants being
disconnected. The application instance however still shows the connections to
the room at time of this failure. The problem starts to compound itself as
people try to access the room ( connect to the application instance ). After
the start of this problem when we look in the admin console the live log for
the application instance does not show any new connection attempts nor accept
or failure to connect but the application instance registers the connection
against the total connect to the application instance. People are unable to
connect to this room. As they continue to try they increase the connection
number registered for this application instance until the point that the
application instance total connected takes us past our license limit. The
application instance seems to register and collect the connections even though
there is no connection to the application instance. We then start to see the
logging of the license limit exceeded.

Please note that there are no error/warning messages generated in logs that
indicate anything has failed. We can only determine that a failure once we
start to experience the symptoms described for the application instance.

We have experienced this issue in the past with fcs. It was not until the last
update of fcs that the probem went away. However, fms 2.0.0 onward has always
had this issue for us.

The problem with this issue is that it is difficult to create and study. We
can go 1 hour and crash or we may have success for over two weeks.

In an attempt to search for a solution to the problem we have started
focussing on this comment in Adobe ?TechNote: http://www.yqcomputer.com/

?In Server.xml, find the ResourceLimits node (around line 181). It contains
the Connector node (around line 203). Within the HTTP node (around line 205),
create a new line and insert MaxConnectionThreads and give it an appropriate
value:
<MaxConnectionThreads>20</MaxConnectionThreads>
The default is 10.?

We have found that by adding the change to our HTTP node that we have lessened
the frequency of our problem. Also, we have taken all remoting away from the
time of connection. We now accept all connections and then do our work after
connection.



Another step we have taken to minimize the impact of this problem is to define
the scope as ?inst? in the application.xml. By creating a separate fmscore.exe
for each application instance it gives us the ability to correct a problem with
a faulting application instances without disturbing our entire server traffic.

Any further insight into this problem would be greatly appreciated.


Regards,

iGx

Hi Dean,

good to hear you liked my technote, but please adjust this setting very
carefully - this setting wasn't originally documented, in my opinion for good
reasons.

One thing to note: All connections count against you license limit.

If this technote helped, it may be the case that your FR calls take quite a
long time and they pile up. You don't see these connections in the admin
console. Use netstats or an equivalent system command to do further
investigation.

Check on the connection lifetime of your outbound calls - maybe you can do
some performance tuning on the backend.

Have you checked out the admin API to get more information from the server?

Hi Dean,

I may have a solution to your problem - go contact me directly. I've left a private message in the forum with my email address.

As indicated by the ongoing discussion, this issue can have different causes -
can you describe your application a little bit? How many users are connecting?
What kind of license do you have?
Do you have server-side code? If yes, what is it doing?
On what OS are you running?
What's the exact version (e.g. 2.0.3 r68) you're using?

Hello RUND,

We are using fms 2.0.3 at the moment and have found some success with making
some changes to the config files. As noted earlier in the forum post we have
added <MaxConnectionThreads>20</MaxConnectionThreads> to our server.xml. Also,
we are finding it helpful to change the JSEngine setting in the application.xml
to a setting of 61440.

Hope that helps.

Regards,

Dean

[B]What kind of license do you have?[/B]
1000-40 stacked (2000-80)
[B]What's the exact version (e.g. 2.0.3 r68) you're using?[/B]
2.0.2 AND 2.0.3 (revision unknown, but the problem happened on both updates)
[B]On what OS are you running?[/B]
Win2K3 Standard
[B]Do you have server-side code?[/B]
Yes
[B]If yes, what is it doing?[/B]
A CHILD FM Server connects to a PARENT FM Server (on an internal LAN) and
subscribes to its live streams. External clients then connect to the CHILD
Servers and request these streams. We generally have ~1000 active connections
to a CHILD server at any given time. The clients generally connect and
disconnect frequently but we never exceed our license of 2000 simultaneous
connections or 80 Mbps bandwidth. The License.Limit.Exceeded usually starts
occurring within minutes. Any suggestions?

Hi JW,

can you cross-check the overall connections, including all conneciton types?
With a minimum of 1000 connections to your child servers at any given time it
does not sound improbable that you're hitting a true license limit.

In general FMS doesn't really make a mistake when counting the license limits.
Please be aware, that all connections count against your license limit -
including HTTP, WebService, and RPC/FR calls.

Please do some investigation on connections and their state - It should help
to use netstat and the admin API to get more information about what's happening
on your servers.

The reason I am saying this is: In some revisions, and that's why I keep
asking for the exact version numbers, there is a problem as FMS is counting
rejected connections. Other customers have just wrong configurations and
unused connections are staying open for too long.

@ patrick:
It's a multi-room, multi-server videochat system
[b]How many users are connecting?[/b]
50-60 connections per server
[b]What kind of license do you have?[/b]
When i first replied we used 1000-40.pro on each server (4xwin2k3 dual
processor), now we're back to 150.pro, we decided we were never gonna reach
1000 connections since our bandwidth consumption is too high for that.
[b]Do you have server-side code? If yes, what is it doing?[/b]
Yes, authentication, ewallet system, chats, proxied so's between servers and
application instances.
[b]On what OS are you running?[/b]
win2k3 server
[b]What's the exact version (e.g. 2.0.3 r68) you're using?[/b]
2.0.3 r68
[b]by the way: I am sorry, but I can not reply to your private messages,
because you're declining them.[/b]
and i'm sorry too, but now i'm now accepting them...

I allready did what iGx advised:
<MaxConnectionThreads>20</MaxConnectionThreads> in our server.xml. And I
changed the JSEngine setting in the application.xml to a setting of 61440. And
it all seemed to work, untill last night. Again the license limit on
connections :(

I'm getting pretty desperate now, the client even suggested calling in extra
fms help. I don't know about what mm/adobe can do for us, but any help is
appreciated!

Hello RUNd,

You can minimize the impact of this problem by changing the scope in the
application.xml to "inst". If each chatroom is an application instance fms will
create an fmscore.exe for each room. Therefore, if you encounter this problem
you only lose a room and not your entire fms service. When you have identified
a problem room go to the log file for that room and look at application.00.log.
The last entry will tell you what core PID is associated with the locked room.
Open your task manager on the fms server, selet processes and if Process ID is
not shown click VIEW/select columns and check Process ID. Once you can see the
fmscore.ext PID end the process using task manager. Shortly thereafter we find
that fms will correct the problem and restart the room with a new application
instance and core.

I have heard that someone has had success with fms 2.0.1 with this issue.
Although we had tried that in the past we are testing fms 2.0.1 with our
customized server config xml files and improved flash remoting. We hope will
show some results. I will post here if we have any success.

Regards,

iGx

Hello,

We have noted a bit of an interesting pattern with all the cores that are
associated with applications that seem to hang. When we view the admin logs we
find the following for a given core and its associated application instance.

-2006-11-15 11:06:04 17316 (i)2581241 Connection to admin received.
-2006-11-15 11:06:04 17316 (i)2581243 Connection from core 18756 received.
-2006-11-15 11:06:04 17316 (i)2581244 Connection from core 18756 accepted.
-2006-11-15 11:06:13 17316 (i)2581242 Core (18756) disconnected :
_defaultRoot_:_defaultVHost_:chat:room221.

However, our problem cores are logged as such:

-2006-11-15 12:37:16 4732 (i)2581241 Connection to admin received. -2006-11-15
12:37:16 4732 (i)2581243 Connection from core 5328 received. -2006-11-15
12:37:16 4732 (i)2581244 Connection from core 5328 accepted. -

* note: there is no disconnect for Core 5328

It is not until we manually remove the core using taskmanger after identifying
a problem wth the assocated application instance do we see the following
disconnect message that would typically follow a normal startup in the admin
log.

-2006-11-15 21:09:30 4732 (i)2581242 Core (5328) disconnected :
_defaultRoot_:_defaultVHost_:chat:room111.

Does anyone else see this type of break in pattern in their log file?

Thanks!

iGx