[jawsscripts] Complete JAWS 14/15 UIAScriptAPI interface spec (was Re: Re: Scripting Avaya One-X Agent)
- From: Doug Lee <doug.lee@xxxxxxxxxxxxxxxx>
- To: jawsscripts@xxxxxxxxxxxxx
- Date: Fri, 8 Nov 2013 18:48:26 -0500
Just for completeness, here's my summary of the UIAScriptAPI DLL COM
interface for JAWS 14 and 15, with info about what each of those JAWS
versions allows. Though I see no published examples in source form in
JAWS 14, the following JSS files in JAWS 15 (I'm looking at a public
beta here) contain uses of these objects: Office.jss,
TWinUI touch.jss, Windows.UI.Search.jss, touch.jss. A search for
"UIAScriptAPI" in Office.jss quickly yields a pretty nifty tree-filter
example, which I will paste in its entirety below the DLL interface
summary. To find that example without reading through the interface
spec, search past this line for "beta" and you should find the comment
I put right above the Office.jss function.
Thanks, Soronel, for helping bring this interface to my attention. It
looks rather complex but very powerful, and relevant to new
applications.
All items start in JAWS 14 unless noted differently in the margin.
Member and other counts reflect JAWS 15. "14!" means something in JAWS
14 was removed in JAWS 15. "**" in the margin indicates a note, which
appears below the line, indented.
Output from Lib UIAScriptAPI.dll:
HasDiskImage Win32 TypeLib UIAScriptAPILib version 1.0
CoClasses 17, Interfaces 18, TypeInfos 35
15 dispinterface IFSUIAEvents Member count 3
15 eventFunction AutomationEvent(Dispatch element, I4 eventID) as HResult
15 eventFunction FocusChangedEvent(Dispatch element) as HResult
15 eventFunction PropertyChangedEvent(Dispatch element, I4 propertyID,
Variant newValue) as HResult
dispinterface IFSUIA Member count 22
15 function AddAutomationEventHandler(I4 eventID, Dispatch element, I4
treeScope) as Bool
15 function AddFocusChangedEventHandler as Bool
15 function AddPropertyChangedEventHandler(I4 propertyID, Dispatch element,
I4 treeScope) as Bool
function CompareElements(Dispatch disp1, Dispatch disp2) as Bool
function CreateAndCondition(Dispatch disp1, Dispatch disp2) as Dispatch
function CreateBoolPropertyCondition(I4 propertyID, I4 value) as Dispatch
function CreateContentViewCondition as Dispatch
function CreateControlViewCondition as Dispatch
function CreateIntPropertyCondition(I4 propertyID, I4 value) as Dispatch
function CreateNotCondition(Dispatch disp) as Dispatch
function CreateOrCondition(Dispatch disp1, Dispatch disp2) as Dispatch
function CreateRawViewCondition as Dispatch
function CreateStringPropertyCondition(I4 propertyID, BStr value) as
Dispatch
function CreateTreeWalker(Dispatch disp) as Dispatch
14! ReadProperty Events as Dispatch
function GetElementFromHandle(I4 handle) as Dispatch
function GetElementFromIAccessible(Dispatch disp, I4 childID) as Dispatch
function GetElementFromPoint(I4 x, I4 y) as Dispatch
function GetFocusedElement as Dispatch
function GetRootElement as Dispatch
15 function RemoveAutomationEventHandler(I4 eventID, Dispatch element) as
Bool
** function RemoveFocusChangedEventHandler as Bool
(no documented return type until JAWS 15)
15 function RemovePropertyChangedEventHandler(Dispatch element) as Bool
dispinterface IFSUIACondition Member count 0
dispinterface IFSUIATreeWalker Member count 6
ReadProperty Condition as Dispatch
Read/WriteProperty currentElement as Dispatch
function GoToFirstChild as Bool
function GoToNextSibling as Bool
function GoToParent as Bool
function GoToPriorSibling as Bool
dispinterface IFSUIAElement Member count 45
ReadProperty acceleratorKey as BStr
ReadProperty accessKey as BStr
ReadProperty ariaProperties as BStr
ReadProperty ariaRole as BStr
ReadProperty automationId as BStr
function BoundingRectangle(pointer to I4 left, pointer to I4 top,
pointer to I4 right, pointer to I4 bottom) as Bool
function BuildUpdatedCache as Dispatch
ReadProperty className as BStr
ReadProperty controlType as I4
function FindAll(I4 treeScope, Dispatch dispCondition) as Dispatch
function FindFirst(I4 treeScope, Dispatch dispCondition) as Dispatch
ReadProperty FlowsTo as Dispatch
function GetClickablePoint(pointer to I4 x, pointer to I4 y) as Bool
function GetExpandCollapsePattern as Dispatch
function GetGridItemPattern as Dispatch
function GetGridPattern as Dispatch
function GetInvokePattern as Dispatch
function GetLegacyIAccessiblePattern as Dispatch
function GetPropertyValue(I4 propertyID) as Variant
function GetRangeValuePattern as Dispatch
function GetScrollItemPattern as Dispatch
15 function GetScrollPattern as Dispatch
function GetSelectionItemPattern as Dispatch
function GetTextChildPattern as Dispatch
function GetTogglePattern as Dispatch
function GetValuePattern as Dispatch
ReadProperty hasKeyboardFocus as Bool
ReadProperty helpText as BStr
ReadProperty IsContentElement as Bool
ReadProperty IsControlElement as Bool
ReadProperty isDataValidForForm as Bool
ReadProperty isEnabled as Bool
ReadProperty isKeyboardFocusable as Bool
ReadProperty isOffscreen as Bool
ReadProperty isPassword as Bool
ReadProperty isRequiredForForm as Bool
ReadProperty itemStatus as BStr
ReadProperty itemType as BStr
ReadProperty localizedControlType as BStr
ReadProperty name as BStr
ReadProperty nativeWindowHandle as I4
ReadProperty orientation as I4
ReadProperty processId as I4
ReadProperty providerDescription as BStr
function SetFocus as Bool
dispinterface IFSUIAExpandCollapse Member count 3
sub Collapse
sub Expand
ReadProperty ExpandCollapseState as I4
dispinterface IFSUIAGridItem Member count 5
ReadProperty column as I4
ReadProperty columnSpan as I4
function ContainingGrid as Dispatch
ReadProperty row as I4
ReadProperty rowSpan as I4
dispinterface IFSUIAGrid Member count 2
ReadProperty columnCount as I4
ReadProperty rowCount as I4
dispinterface IFSUIAInvoke Member count 1
function Invoke as Bool
dispinterface IfsUIARangeValue Member count 7
ReadProperty isReadOnly as Bool
ReadProperty largeChange as I4
ReadProperty maximum as I4
ReadProperty minimum as I4
15 function SetValue(I4 value) as Bool
ReadProperty smallChange as I4
ReadProperty value as I4
dispinterface IFSUIASelectionItem Member count 4
15 function AddToSelection as Bool
ReadProperty isSelected as Bool
15 function RemoveFromSelection as Bool
sub Select
dispinterface IFSUIAToggle Member count 2
15 sub Toggle
ReadProperty toggleState as I4
dispinterface IFSUIAValue Member count 2
ReadProperty isReadOnly as Bool
ReadProperty value as BStr
dispinterface IFSUIAElementArray Member count 2
function GetElement(I4 index) as Dispatch
ReadProperty Length as I4
dispinterface IFSUIAScrollItem Member count 1
sub ScrollIntoView
dispinterface IFSUIATextChild Member count 0
dispinterface IFSUIALegacyIAccessible Member count 14
ReadProperty childID as I4
ReadProperty DefaultAction as BStr
ReadProperty Description as BStr
sub DoDefaultAction
function GetIAccessible as Dispatch
function GetSelection as Dispatch
ReadProperty Help as BStr
ReadProperty KeyboardShortcut as BStr
ReadProperty name as BStr
ReadProperty Role as I4
function Select(I4 selectionFlags) as Bool
function SetValue(BStr value) as Bool
ReadProperty State as I4
ReadProperty value as BStr
14!dispinterface _IFSUIAEventsEvents Member count 1
14! eventFunction FocusChangedEvent(Dispatch element) as HResult
15 dispinterface IFSUIAScroll Member count 8
15 ReadProperty HorizontallyScrollable as Bool
15 ReadProperty HorizontalScrollPercent as I4
15 ReadProperty HorizontalViewSize as I4
15 function Scroll(I4 horizontal, I4 vertical) as Bool
15 function SetScrollPercent(I4 horizontal, I4 vertical) as Bool
15 ReadProperty VerticallyScrollable as Bool
15 ReadProperty VerticalScrollPercent as I4
15 ReadProperty VerticalViewSize as I4
; From JAWS 15 (public beta) Office.jss:
Int Function SayBackStageHelpMessage (handle hWnd)
Var
String sGroupName = GetObjectName (TRUE, 1),
Object oUIAMain = CreateObjectEx ("FreedomSci.UIA", FALSE,
"UIAScriptAPI.x.manifest"),
Object oObjectNameCondition = oUIAMain.CreateStringPropertyCondition
(UIA_NamePropertyId, sGroupName),
Object oGroupCondition = oUIAMain.CreateAndCondition
(oObjectNameCondition, oUIAMain.CreateIntPropertyCondition
(UIA_ControlTypePropertyId, UIA_GroupControlTypeId)),
Object oTextCondition = oUIAMain.CreateAndCondition
(oUIAMain.CreateNotCondition (oObjectNameCondition),
oUIAMain.CreateIntPropertyCondition (UIA_ControlTypePropertyId,
UIA_TextControlTypeId)),
Object oElement = oUIAMain.GetElementFromHandle (hWnd),
Object oGroups,
Object oStatic,
Int i
If oUIAMain ; main UIA object was created successfully...
&& oObjectNameCondition ; object name condition was created successfully...
&& oGroupCondition ; group condition created successfully...
&& oTextCondition ; text condition created successfully...
&& oElement ; element object created successfully....
oGroups = oElement.FindAll (TreeScope_Descendants, oGroupCondition)
If oGroups ; array containing groups is created successfully...
&& oGroups.Length ; and length is more then zero...
oStatic = oGroups.GetElement (0).FindAll (TreeScope_Children,
oTextCondition)
For i = 0 to oStatic.Length
Say (oStatic.GetElement (i).Name, OT_SCREEN_MESSAGE)
EndFor
Return (TRUE)
EndIf
EndIf
Return (FALSE)
EndFunction
On Fri, Nov 08, 2013 at 01:20:28PM -0900, Soronel Haetir wrote:
A followup to my last post, the correct progid is "FreedomSci.UIA"
(not FreedomSci.FSUia") and the manifest name is
:"UIAScriptAPI.x.manifest (not "UIAScriptAPI.exe.manifest").
I have confirmed that with the correct progid/manifest pair that jaws'
CreateObjectEx is able to acquire an object.
On 11/8/13, Soronel Haetir <soronel.haetir@xxxxxxxxx> wrote:
> Looking at the manifest embedded in AccessibleObjectTools.dll,
> "FreedomSci.AccessibleTree" is the only comClass element with a progId
> attribute.
>
> There is also an IDL style definition for IAccessibleObject in
> HomerowUIAObject.jss in the shared scripts directory.
>
> The only other file I was able to find in the jaws progream directory
> that contains an embedded type library is UIAScriptAPI.dll, which can
> be used to create objects with progid "FreedomSci.UIA".
>
> On 11/8/13, Jim Bauer <holdsworthfan@xxxxxx> wrote:
>> If you did, I never saw it. :-) I for one would find a summary of this
>> lib useful.
>>
>>
>> On 11/8/2013 12:14 PM, Doug Lee wrote:
>>> My first method of doing this was to find the DLL and open it in
>>> Word's VBA COM viewer. More recently I wrote something that creates a
>>> summary of a COM lib's properties and methods. It's not perfect and I
>>> didn't write it in a very easily portable way, but sometimes I've used
>>> it to produce a usage summary to post here. I think I already did that
>>> for this lib, but if not and you need it, write back here and I'll
>>> post one to this list again.
>>>
>>> On Fri, Nov 08, 2013 at 11:27:24AM -0500, John Robichaud wrote:
>>> Doug,
>>> How do you find the COM typelib information for the accessibleTree?
>>> John
>>>
>>> -----Original Message-----
>>> From: jawsscripts-bounce@xxxxxxxxxxxxx
>>> [mailto:jawsscripts-bounce@xxxxxxxxxxxxx] On Behalf Of Doug Lee
>>> Sent: Tuesday, October 29, 2013 5:49 AM
>>> To: jawsscripts@xxxxxxxxxxxxx
>>> Subject: [jawsscripts] Re: Scripting Avaya One-X Agent
>>>
>>> I found the VS FS code for getting an accessibleObjectTree and wrote
>>> code
>>> based on that to get one of those. Then I examined the COM typelib
>>> information for those objects to find out what I could do with them.
>>>
>>> On Tue, Oct 29, 2013 at 09:13:23AM +0200, Stefan Moisei wrote:
>>> It's very unfortunate that UIA scripting is not documented at all. An
>>> increasing number of applications use it. Ironically, I think NVDA
>>> provides
>>> a better scripting support for it, once you get past the initial
>>> details.
>>> Still, Doug, did you use only the functions found in visual studio
>>> scripts,
>>> or did you find about others from some UIA interfaces?
>>> Thanks.
>>> www.Vortex.IM
>>> -----Original Message-----
>>> From: Doug Lee
>>> Sent: Saturday, October 26, 2013 11:37 AM
>>> To: jawsscripts@xxxxxxxxxxxxx
>>> Subject: [jawsscripts] Re: Scripting Avaya One-X Agent
>>>
>>> When I coded that part of BX, I used FS scripts for, I believe, Visual
>>> Studio as a reference.
>>>
>>> On Sat, Oct 26, 2013 at 09:22:02AM +0200, Bram Duvigneau wrote:
>>> John,
>>> I've had some discussion about this earlier on the list when I was
>>> struggling with the MSAA functions and getting no usable results. The
>>> idea
>>> to use this COM object came from the BX Utilities source code.
>>> However, I still haven't found any official documentation what so ever
>>> about
>>> this mechanism, which is to bad because we'll need it more and more in
>>> the
>>> future.
>>>
>>> Bram
>>> On 26-10-2013 3:53, John Robichaud wrote:
>>>> Bram,
>>>> In your example, how did you determine the COM class for the first
>>>> parameter and the manifest file for the third parameter in
>>>> CreateObject Ex?
>>>> John Robichaud
>>>>
>>>> -----Original Message-----
>>>> From: jawsscripts-bounce@xxxxxxxxxxxxx
>>>> [mailto:jawsscripts-bounce@xxxxxxxxxxxxx] On Behalf Of Bram Duvigneau
>>>> Sent: Friday, October 25, 2013 3:01 AM
>>>> To: jawsscripts@xxxxxxxxxxxxx
>>>> Subject: [jawsscripts] Re: Scripting Avaya One-X Agent
>>>>
>>>> Hello,
>>>> I recently scripted another Avaya product and only got access to the
>>>> controls through UIA. Please see the simple function below that I used
>>>> to display incoming call notifications in a message box:
>>>>
>>>> Void Function HandleIncomingCall ()
>>>> var object builder = createObjectEx("FreedomSci.AccessibleTree",
>>>> False, "AccessibleObjectTools.x.manifest"),
>>>> object o = builder.Build(hCallWindow),
>>>> String title,
>>>> String callType,
>>>> String caller,
>>>> String message
>>>>
>>>> let o = o.firstChild
>>>>
>>>> let title = FormatString("%1", o.Name)
>>>>
>>>> let o = o.nextSibling
>>>> let o = o.nextSibling
>>>> let o = o.nextSibling
>>>> let callType = o.Name
>>>>
>>>> let o = o.nextSibling
>>>> let caller = o.Name
>>>>
>>>> let message = FormatString("%1: %2", callType, caller)
>>>>
>>>> ExMessageBox (message, title, MB_OK) EndFunction
>>>>
>>>>
>>>> Good luck,
>>>>
>>>> Bram
>>>> On 25-10-2013 6:18, Jitendra wrote:
>>>>> Well, I don't know if you can get any fruit out of it, but Dlee.org
>>>>> and download the audio player scripts from there, and look the time
>>>>> module code, if you have several windows, you may reach them, and for
>>>>> MSAA, you may download the skype scripts, and examine the code to
>>>>> work with objects.
>>>>> if it does not help, and the software is trial or free to download, I
>>>>> can try helping you.
>>>>> Thanks, Jitendra, India.
>>>>> John Robichaud wrote:
>>>>>> Wondering if anyone has attempted scripting Avaya One-X Agent
>>>>>> version
>>>> 2.5.
>>>>>> This is a telphone control system.
>>>>>> Although Avaya's controls are spoken accurately when tabbing through
>>>>>> the interface, I've been unsuccessful in creating scripts to provide
>>>>>> direct access to each control.
>>>>>>
>>>>>> Using Jaws script utility mode, all controls exhibit the same handle
>>>>>> and control ID. Class throughout is given as
>>>>>> HwndWrapper[OneXAgentUI.exe;;7b95d218-e604]. Utility mode does give
>>>>>> an unique Object ID, Object Name, and Object Type for each control
>>>>>> but I've not been able to use that info to reach the controls. Can
>>>>>> anyone suggest an approach?
>>>>>>
>>>>>> John Robichaud
>>>>>>
>>>>>>
>>>>>>
>>>>>> __________???
>>>>>>
>>>>>> View the list's information and change your settings at
>>>>>> http://www.freelists.org/list/jawsscripts
>>>>>>
>>>>>>
>>>>>>
>>>>> --- Disclaimer --- The information in this mail is confidential and
>>>>> is intended solely for addressee. Access to this mail by anyone else
>>>>> is unauthorised. Copying or further distribution beyond the original
>>>>> recipient may be unlawful. Any opinion expressed in this mail is that
>>>>> of sender and does not necessarily reflect that of State Bank group.
>>>>> --- __________???
>>>>>
>>>>> View the list's information and change your settings at
>>>>> http://www.freelists.org/list/jawsscripts
>>>>>
>>>>
>>>>
>>>> __________???
>>>>
>>>> View the list's information and change your settings at
>>>> http://www.freelists.org/list/jawsscripts
>>>>
>>>> __________???
>>>>
>>>> View the list's information and change your settings at
>>>> http://www.freelists.org/list/jawsscripts
>>>>
>>>
>>>
>>>
>>> __________???
>>>
>>> View the list's information and change your settings at
>>> http://www.freelists.org/list/jawsscripts
>>>
>>> --
>>> Doug Lee, Senior Accessibility Programmer SSB BART Group -
>>> Accessibility-on-Demand mailto:doug.lee@xxxxxxxxxxxxxxxx
>>> http://www.ssbbartgroup.com "While they were saying among themselves it
>>> cannot be done, it was done." --Helen Keller __________???
>>>
>>> View the list's information and change your settings at
>>> http://www.freelists.org/list/jawsscripts
>>>
>>> __________???
>>>
>>> View the list's information and change your settings at
>>> http://www.freelists.org/list/jawsscripts
>>>
>>> --
>>> Doug Lee, Senior Accessibility Programmer SSB BART Group -
>>> Accessibility-on-Demand mailto:doug.lee@xxxxxxxxxxxxxxxx
>>> http://www.ssbbartgroup.com "While they were saying among themselves it
>>> cannot be done, it was done." --Helen Keller __________o?=
>>>
>>> View the list's information and change your settings at
>>> http://www.freelists.org/list/jawsscripts
>>>
>>> __________???
>>>
>>> View the list's information and change your settings at
>>> http://www.freelists.org/list/jawsscripts
>>>
>>
>> __________???
>>
>> View the list's information and change your settings at
>> http://www.freelists.org/list/jawsscripts
>>
>>
>
>
> --
> Soronel Haetir
> soronel.haetir@xxxxxxxxx
>
--
Soronel Haetir
soronel.haetir@xxxxxxxxx
__________???
View the list's information and change your settings at
http://www.freelists.org/list/jawsscripts
--
Doug Lee, Senior Accessibility Programmer
SSB BART Group - Accessibility-on-Demand
mailto:doug.lee@xxxxxxxxxxxxxxxx http://www.ssbbartgroup.com
"While they were saying among themselves it cannot be done,
it was done." --Helen Keller
__________�
View the list's information and change your settings at
http://www.freelists.org/list/jawsscripts
Other related posts:
- » [jawsscripts] Complete JAWS 14/15 UIAScriptAPI interface spec (was Re: Re: Scripting Avaya One-X Agent) - Doug Lee