-
Notifications
You must be signed in to change notification settings - Fork 27
Expand file tree
/
Copy pathmonitoring.html
More file actions
399 lines (377 loc) · 22.6 KB
/
Copy pathmonitoring.html
File metadata and controls
399 lines (377 loc) · 22.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<!--
Copyright (c) 1997, 2020 Oracle and/or its affiliates. All rights reserved.
This program and the accompanying materials are made available under the
terms of the Eclipse Public License v. 2.0 which is available at
http://www.eclipse.org/legal/epl-2.0, or the Eclipse Distribution License
v. 1.0 which is available at
http://www.eclipse.org/org/documents/edl-v10.php.
This Source Code may also be made available under the following Secondary
Licenses when the conditions for such availability set forth in the Eclipse
Public License v. 2.0 are satisfied: GNU General Public License v2.0
w/Classpath exception which is available at
https://www.gnu.org/software/classpath/license.html.
SPDX-License-Identifier: EPL-2.0 OR BSD-3-Clause OR GPL-2.0 WITH
Classpath-exception-2.0
-->
<HTML>
<HEAD>
<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=iso-8859-1">
<TITLE>monitoring</TITLE>
<META NAME="GENERATOR" CONTENT="StarOffice 7 (Linux)">
<META NAME="CREATED" CONTENT="20041117;16034300">
<META NAME="CHANGEDBY" CONTENT="Ken Cavanaugh">
<META NAME="CHANGED" CONTENT="20041118;14294200">
</HEAD>
<BODY LANG="en-US" DIR="LTR" BACKGROUND="images/backgrounds/p12c08.gif">
<H2>ORB Monitoring Extensions for AS 8.2</H2>
<P>It has become apparent that some changes are needed in the ORB
monitoring framework for AS 8.2. The current monitoring code in the ORB is
overly long and difficult to construct. We also need to consider upcoming
requirements in AS 8.2 in areas such as dynamic configuration changes and
self-tuning. I would like to build a dynamic MBean directly in the ORB so that
we can more readily experiment with different maangement tools, and also to
fully validate the ORB monitoring framework design. </P>
<P>Note: this is basically obsolete, given the new JMX utilities</P>
<H3>Simplifying the Usage of ORB Monitoring</H3>
<P>Currently the ORB monitoring code is rather long. For example, here is
the code in CorbaInboundConnectionCacheImpl: </P>
<PRE> protected void registerWithMonitoring()
{
// ORB
MonitoredObject orbMO =
orb.getMonitoringManager().getRootMonitoredObject();
// REVISIT - add ORBUtil mkdir -p like operation for this.
// CONNECTION
MonitoredObject connectionMO =
orbMO.getChild(MonitoringConstants.CONNECTION_MONITORING_ROOT);
if (connectionMO == null) {
connectionMO =
MonitoringFactories.getMonitoredObjectFactory()
.createMonitoredObject(
MonitoringConstants.CONNECTION_MONITORING_ROOT,
MonitoringConstants.CONNECTION_MONITORING_ROOT_DESCRIPTION);
orbMO.addChild(connectionMO);
}
// INBOUND CONNECTION
MonitoredObject inboundConnectionMO =
connectionMO.getChild(
MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT);
if (inboundConnectionMO == null) {
inboundConnectionMO =
MonitoringFactories.getMonitoredObjectFactory()
.createMonitoredObject(
MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT,
MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT_DESCRIPTION);
connectionMO.addChild(inboundConnectionMO);
}
// NODE FOR THIS CACHE
MonitoredObject thisMO =
inboundConnectionMO.getChild(getMonitoringName());
if (thisMO == null) {
thisMO =
MonitoringFactories.getMonitoredObjectFactory()
.createMonitoredObject(
getMonitoringName(),
MonitoringConstants.CONNECTION_MONITORING_DESCRIPTION);
inboundConnectionMO.addChild(thisMO);
}
LongMonitoredAttributeBase attribute;
// ATTRIBUTE
attribute = new
LongMonitoredAttributeBase(
MonitoringConstants.CONNECTION_TOTAL_NUMBER_OF_CONNECTIONS,
MonitoringConstants.CONNECTION_TOTAL_NUMBER_OF_CONNECTIONS_DESCRIPTION)
{
public Object getValue() {
return new Long(CorbaInboundConnectionCacheImpl.this.numberOfConnections());
}
};
thisMO.addAttribute(attribute);
// ATTRIBUTE
attribute = new
LongMonitoredAttributeBase(
MonitoringConstants.CONNECTION_NUMBER_OF_IDLE_CONNECTIONS,
MonitoringConstants.CONNECTION_NUMBER_OF_IDLE_CONNECTIONS_DESCRIPTION)
{
public Object getValue() {
return new Long(CorbaInboundConnectionCacheImpl.this.numberOfIdleConnections());
}
};
thisMO.addAttribute(attribute);
// ATTRIBUTE
attribute = new
LongMonitoredAttributeBase(
MonitoringConstants.CONNECTION_NUMBER_OF_BUSY_CONNECTIONS,
MonitoringConstants.CONNECTION_NUMBER_OF_BUSY_CONNECTIONS_DESCRIPTION)
{
public Object getValue() {
return new Long(CorbaInboundConnectionCacheImpl.this.numberOfBusyConnections());
}
};
thisMO.addAttribute(attribute);
}</PRE>
<P> Fully formatted, this is 82 lines of source. Here are some thoughts on
simplifying assumptions: </P>
<OL>
<LI>
<P>(BIG assumption): Use a Dynamic MBean constructed by introspection
on the MonitoredXXX objects reachable from the Root MonitoredObject in the ORB.
This eliminates the need for MonitoringConstants, and consequently simplifies
the declarations. </P>
<P STYLE="margin-bottom: 0in">There is a problem with this: the app
server currently has a surprisingly complicated way of getting from ORB
MonitoredObject (which is dynamic) to a Dynamic MBean by going through a static
Stats layer (ugh). We could simply keep the necessary constants for the AS in
MonitoringConstants, but I would rather get rid of them. </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">Any reasonable name (let's say a Java
identifier) is legal for MonitoredObjects. </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">MonitoredAttribute names must start with
a capital letter and be Java identifiers (SomethingLikeThis). </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">All descriptions for MonitoredObjects and
MonitoredAttributes are stored in a resource file (Can we make this dynamic if
necessary?) </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">The key in the resource file for object
or attribute "XXX" is "XXX.description". </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">The MonitoredAttribute SomethingLikeThis
is implemented by a method named somethingLikeThis. RO means only a getter
(TYPE somethingLikeThis()) while a RW attribute has a setter as well (void
somethingLikeThis(TYPE)). </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">The TYPE of an attribute is determined
from the getter/setter methods. </P> </LI>
<LI>
<P>We provide a utility class MUtil in spi.monitoring. </P> </LI>
</OL>
<P>With these assumptions, we could get the example above down to: </P>
<PRE>protected void registerWithMonitoring()
{
MonitoredObject thisMO = MUtil.get( orb, "/Connections/Inbound/" + getMonitoringName() ) ;
thisMO.addAttribute( "NumberOfConnections", this ) ;
thisMO.addAttribute( "NumberOfIdleConnections", this ) ;
thisMO.addAttribute( "NumberOfBusyConnections", this ) ;
}</PRE>
<P> That's it! Just 4 lines of code. </P>
<P>Here the get method get or creates the MonitoredObjects as needed (as
Harold noted, a "mkdir -p" like operation). addAttribute uses all of the
assumptions mentioned above to derive the appropriate information. </P>
<H4>MonitoredObjects for JavaBeans</H4>
<P>A further extension to this is possible. Consider the case of ORBData.
ORBData has a lot of methods that look like: </P>
<PRE STYLE="margin-bottom: 0.2in"> (some data type) getXXXX() ;</PRE>
<P> This is the standard pattern for a JavaBean accessor method for a
property named "XXXX". In addition, we have methods like: </P>
<PRE> boolean isJavaSerializationEnabled() ;
boolean showInfoMessages() ;</PRE>
<P> A method name that returns a boolean and starts with "is" is a JavaBean
accessor method. Unfortunately ORBData has a lot of boolean methods that do not
start with "is", so we cannot directly use the JavaBean Introspector. However,
the job is simple enough that that is not necessary. Alternatively we could
pull all of the information out of the ParserTable class. In any case, the idea
is to create a MonitoredObject for ORBData that reflects all of the
configuration data for the ORB. We will discuss why this is useful a bit later.
</P>
<H3>Some ORB Monitoring Problems</H3>
<P>Currently there are two different ways of getting information the root
monitored object: either we call </P>
<PRE>MonitoringFactories.getMonitoringManagerFactory().createMonitoringManager(
(root name), description ).getRootMonitoredObject()</PRE>
<P> or we call </P>
<PRE STYLE="margin-bottom: 0.2in">orb.getMonitoringManager().getRootMonitoredObject()</PRE>
<P> Having two different ways of getting information is a frequent source
of problems. These two mechanisms agree when the (root name) is the same as the
corresponding value in the ORB, but that is currently not the case, due to the
change I made to make sure that the monitoring code does not get overwritten by
multiple ORBs. Hosever, the createMonitoringManager call will create a new
MonitoringManager and root MonitoredObject whenever it is called. Currently I
see two difference MonitoringManager objects in the app server: one with root
named "orb", and one with root named "orb_S1AS-ORB_1", which comes from the app
server's standard ORB id and the change I made in ORB b42 to fix bug 4919770.
</P>
<P>Another problem occurs here because the ThreadPool always uses the first
form to get to the root MonitoredObject, which is indepedent of the ORB. This
means that all ThreadPools, no matter how they are created or where they are
used, will be associated with the root MonitoredObject that corresponds to the
first ORB created with ORB id "". This means that the threadpool monitoring
will work regardless of the ORB id in the current app server. I still cannot
reproduce Sheetal's bug, but it seems unlikely to be a significant problem.
</P>
<P>On the other hand, the connection monitoring goes straight to the ORB to
get the root monitored object. Some suggestions for 8.2: </P>
<OL>
<LI>
<P STYLE="margin-bottom: 0in">There should be only one possible root
MonitoredObject, and a single MonitoringManager. The ORB should create its
monitoring root under this top-level (probably unnamed) root object, using the
current naming convention. </P> </LI>
<LI>
<P STYLE="margin-bottom: 0in">The ThreadPool should continue to be ORB
independent, as it can be used anywhere. However, the threadpool really should
not both be independent of the ORB and have its monitoring registered under the
ORB. Instead, we should register the threadpool under its own root, independent
of the ORB. </P> </LI>
<LI>
<P>We only need one MonitoringManager, which simply provides access to
/ in the monitoring hierarchy. </P> </LI>
</OL>
<P>Current hierarchy (multiple named roots): </P>
<PRE>orb
threadpool
Connections
Inbound
Outbound
orb_XXX_1
orb_XXX_2
etc</PRE>
<P> New hierarchy (this requires app server changes): </P>
<PRE>(root)
threadpoolroot
threadpool_1
threadpool_2
(etc. if we create multiple ThreadPoolManagers)
orbroot
orb_XXX_1
Connections
Inbound
Outbound
(plus more TBD as we work on 8.2 and 9)
orb_XXX_2
orb_YYY_1</PRE>
<P> Note that the old hierarchy is confusing when there are multiple ORBs.
There is no confusion in the new hierarchy. For example, </P>
<PRE STYLE="margin-bottom: 0.2in">orb.getMonitoringManager().getRootMonitoredObject() </PRE>
<P> should just return the ORB's root, which is: </P>
<PRE STYLE="margin-bottom: 0.2in">/orbroot/orb_XXX_n</PRE>
<P> MonitoringFactories.getRootMonitoredObject should just return / (and
this replaces the current MonitoringManagerFactory.createMonitoringManager
call). </P>
<H3>Adding Notifications to ORB Monitoring</H3>
<P>Given the possibility of changing MonitoredAttribute values, it becomes
necessary to consider how the ORB can find out about changes. For this, we need
to extend the ORB monitoring framework with events and event listeners. </P>
<P>Model:</P>
<OL>
<LI>
<P>An attribute set method is called.</P> </LI>
<LI>
<P>The attribute examines the registered listeners and invokes the
attributeChanged method on each listener, giving the listener the
MonitoredObject, the MonitoredAttribute, and the old and new values of the
attribute.</P> </LI>
<LI>
<P>Control returns from the attribute set method.</P> </LI>
</OL>
<P>The other case we need to consider is creating and destroying
MonitoredObjects. Here we want to receive either an ObjectCreated or an
ObjectDestroyed notification on the parent MonitoredObject of the
MonitoredObject that was created or removed. Here the notification just needs
the parent and child MonitoredObjects. </P>
<P>A listener needs to implement the following interfaces as needed:</P>
<PRE>interface AttributeListener {
void attributeChanged( MonitoredObject object, MonitoredAttribute attribute, Object oldValue,
Object newValue ) ;
}
interface ObjectListener {
void objectCreated( MonitoredObject parent, MonitoredObject child ) ;
void objectRemoved( MonitoredObject parent, MonitoredObject child ) ;
} ;
</PRE>
<P>MonitoredObject needs new methods to handle the registration of
listeners. I think it may also be useful to have a scope associated with the
registration, so that a listener can be registered either for events from one
MonitoredObject, from the immediate descendants of a MonitoredObject, or from
all descendants of a MonitoredObject. Since we are still using J2SE 1.4, we
will map these scope values to integer constants. Then we need to add the
following methods to MonitoredObject:</P>
<PRE>interface MonitoredObject {
...
void registerAttributeListener( AttributeListener listener, int scope ) ;
AttributeListener[] attributeListeners() ;
void removeAttributeListener( AttributeListener listener ) ;
void registerObjectListener( ObjectListener listener, int scope ) ;
ObjectListener[] objectListeners() ;
void removeObjectListener( ObjectListener listener ) ;
}
</PRE>
<H3>An Architecture for Dynamic Configuration</H3>
<P>The basic idea here is simple: provide attribute change notification
events and configurations agends that listen for these events. But how exactly
does this work?</P>
<P>First, I think we do not need the complex model that JavaBeans use,
where some events can be vetoed. We still need to decide what the notification
model should be, what events we support, and how those events get translated
into Mbean notifications.</P>
<H3>ORB dynamic MBean Support</H3>
<P>The ORB will probably need to be configurable in where it obtains an
MBean server. On J2SE, the ORB should use
java.lang.management.ManagementFactory.getPlatformMBeanServer. What to do in
J2EE is still TBD. </P>
<P>What we need to define here is how to map the ORB
MonitoredObject/MonitoredAttribute framework into MBeans using the JMX
framework. There are some significant differences in approaches taken in the
two framworks. The JMX approach is more complex, with more features including
such items as relations. The ORB Monitoring Framework only supports a single
hierarchy, which is built in to the definition of the MonitoredObject. The
genesis of the ORB framework is for monitoring (mostly) performance-related
attributes of the system, while JMX is intended to be a fully general
management and monitoring framework.</P>
<P>Part of the intent here is to grow the ORB framework to be more general,
so that it can also support ORB management. Primarily this comes about through
the addition of notifications. So far I have seen little need to add actions,
but that is also easily possible.</P>
<P>There are several problems to solve here:</P>
<OL>
<LI>For each MonitoredObject, there is a dynamic MBean whose attributes
correspond to the MonitoredAttributes of the Monitored Object.</LI>
<LI>The name of the MBean is derived from the nesting of the
MonitoredObjects starting at the root.</LI>
<LI>There is an object in the MBean pacakge that listens for
MonitoredObject events (registered against the root, scoped for all
descendants). This handles events as follows:
<OL>
<LI>For an attributedChanged event, the attribute on the
corresponding MBean is updated. </LI>
<LI>For an objectCreated event, an MBean corresponding to the
MonitoredObject is created and registered with the MBean server.</LI>
<LI>For an objectDestroyed evend, the MBean corresponding to the
destroyed MonitoredObject is de-registered from the MBean server.</LI>
</OL></LI>
</OL>
<H3>Dynamic Tracing in Java Applications</H3>
<P>Solaris 10 has recently introduced the dtrace facility, which provides
extremely useful tools for dignosing and tuning the Solaris OS and C/C++
applications. What about Java? Here I am not just interested in things like the
Java stack. Instead, I want to consider extremely large and complex apps like
the app server, which has many interacting modules, each of which has its own
unique requirements for monitoring. As monitoring is a big push for AS 8.2 and
9, it seems that we should seriously consider what we need to do here. </P>
<P>In fact, there is a project to do exactly this: it's called<A
HREF="jtrace_to_svc2.pdf"> jtrace</A>, which is work being done in Janet
Koenig's group with Sanjay Radia and A. Sundar. More information is available
in <A HREF="http://jplan.sfbay/feature/264">jplan</A>, and it is currently
scheduled for Dolphin (J2SE 7). Unfortunately this work is a very long way (3+
years for J2SE, probably >4 for J2EE) out.</P>
<P>I wonder if we could do something sooner? There are several
possibilities:</P>
<OL>
<LI>Add instrumentation at the ORB SPI level. This is a very specific
case, and somewhat labor-intensive to add. The big question still remains about
what to do for the data collection language: DTrace's D is quite nice and
fairly large.</LI>
<LI>Similar to 1, but add the instrumentation at the Container level (in
the component-based development sense; see <A
HREF="component_notes.html">components</A> for a draft of these ideas).</LI>
<LI>Combine several ideas: java.lang.Instrument in J2SE 5.0, byte-code
instrumentation using ASM or BCEL, remote communications (perhaps using
CORBA?), and the current Dtrace to create more-or-less what is needed. The key
piece here is implementing a custom provider for DTrace (for which there is no
information at present).</LI>
<LI>It might be nice to be able to use MBeans as well with a tracing tool
of some sort.</LI>
</OL> </BODY>
</HTML>