Showing posts with label ui-automation. Show all posts
Showing posts with label ui-automation. Show all posts

Tuesday, October 16, 2018

componentOne windows flexgrid automation using CodedUI C#

Leave a Comment

I have only compiled exe application which i need to automate. It has c1.win.c1flexgrid of ComponentOne as Grid. I am using CodedUI to automate this application.

When using Coded UI Record & Playback, it do not highlight or find any row or column but only outer body of this flexgrid which is grid name. If i try to get its children, it return zero whereas, i can see there are lot of rows and columns in this FlexGrid.

I tried to get current patterns of this control using AutomationElement object and it returned me following two patterns as mentioned below.

It has only two patterns implemented.

  1. LegacyIAccessiblePatternIdentifiers.Pattern
  2. ScrollPatternIdentifiers.Pattern

I would like to know how i can automate this type of grid which do not have implemented any other automation pattern and is third party control.

I will be thankful to you if you can share sample or directions.

Regards,

0 Answers

Read More

Saturday, June 23, 2018

Can I make selenium pause for input and resume on a trigger?

Leave a Comment

I am wondering if selenium can do below? I want to automate only certain parts of the automation flow:

  1. Load a web page (built with angular), submit the form with some predefined inputs
  2. On the next page, automatically fill in some data like earlier, but wait for me to fill in some input on specific input fields (can't hard-code this data)
  3. After this, a trigger (like a button press or key combination; outside of the web page) should carry on with the rest of the automated flow and land in page 3 and 4 and so on.

The only option I am familiar with is to write and run custom JS that modifies form elements, in the browser>inspect>console. For above, I'll have to run different functions on each page. For doing this, I can comment out all but the required function call and run it. I think I cannot select and run only one part of the code (for page 1 for example) from the console.

PS: If any of the strict SO folks think this is not fitting SO, where else is a good (automation focused?) place to ask for finding the right tools for this kind of stuff?

3 Answers

Answers 1

There are several waits available in selenium.

Implicit Wait: During Implicit wait if the Web Driver cannot find it immediately because of its availability, it will keep polling (around 250 milli seconds) the DOM to get the element. If the element is not available within the specified Time an NoSuchElementException will be raised. The default setting is zero. Once we set a time, the Web Driver waits for the period of the WebDriver object instance.

Explicit Wait: There can be instance when a particular element takes more than a minute to load. In that case you definitely not like to set a huge time to Implicit wait, as if you do this your browser will going to wait for the same time for every element.

To avoid that situation you can simply put a separate time on the required element only. By following this your browser implicit wait time would be short for every element and it would be large for specific element.

Fluent Wait: Let’s say you have an element which sometime appears in just 1 second and some time it takes minutes to appear. In that case it is better to use fluent wait, as this will try to find element again and again until it find it or until the final timer runs out.

https://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp

http://toolsqa.com/selenium-webdriver/implicit-explicit-n-fluent-wait/

Answers 2

Note: I have used selenium via python, so solution reflects that.

Oh yeah. It's just a python script. Don't think of it in terms if selenium script. A python script can be easily made to wait for input.

print("Hi!. Script Started") # code to load webpage, automatically fill whatever can be entered x = input("Waiting for manual date to be entered. Enter YES when done.") # Enter the data on page manually. Then come back to terminal and type YES and then press enter. if x == 'YES':     continue_script_here() else:     kill_script_or_something_else() 

Answers 3

Option 1:

This can be easily achieved using Explicit wait. Suppose you want to manually enter data in some field. You can make the selenium wait till the point where the field contains a value(Its "value" attribute is not empty). Ex:

WebDriverWait wait = new WebDriverWait(driver, 100); //whatever time you think is sufficient for manually entering the data. WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id(>someid>))); if(ExpectedConditions.attributeToBeNotEmpty(element,"value")) {   //continue with the automation flow } 

Option 2:

It is kinda hacky. What you can do is, At the beginning of the execution open another tab and then switch back to your original one, like this:

driver.findElement(By.cssSelector("body")).sendKeys(Keys.CONTROL +"t"); driver.switchTo().defaultContent(); 

Now execution will start and at the point where you want the script to stop for you to manually enter your data, retrieve all the tabs from selenium in an infinite loop like this-

for(int i =1;i>0;i++) {  ArrayList<String> tabs = new ArrayList<String>(driver.getWindowHandles());  if(tabs.size()==2)  {   //keep checking for the point when the number of tabs becomes 1 again.   continue;  }  else  {   break;  } } //your rest of the automation code 

The idea is to make selenium pause the execution(since it will be stuck in the loop) till the point where number of tabs again become 1. During this, you enter your data and close the empty tab so that the selenium can continue its execution.

You can also try this.

Read More

Saturday, February 3, 2018

Any way to block/remove keyboard hook in uncontrolled window's Menubar?

Leave a Comment

I'm sending keystrokes to an inactive Adobe Flash Projector window with PostMessage, that part works perfectly. I leave it running in the background and it interferes very little with my normal computer usage, which is the intent. The problem comes when I programmatically send the W (or less frequently Q) key while I happen to be holding Ctrl intended for a different windows shortcut. This triggers Ctrl-Q or Ctrl-W, both of which immediately close the Adobe Flash Projector. Ctrl-F and Ctrl-O have some undesirable effects as well.

EDIT: I'm not interested in solutions that briefly release the Ctrl key.

Is there anyway I can unhook shortcut keys from a third party window? It uses a standard OS menubar across the top of the window which is where the shortcuts are listed, so surely there's a way to reassign, unassign, or block it, right?

In the past I tried using these dlls to break the menu. It made it disappear but didn't break the shortcuts.

DllCall("SetMenu", uint, this.id, uint, 0) hMenu := DllCall("GetSystemMenu", "UInt",this.id, "UInt",1) DllCall("DestroyMenu", "Uint",hMenu) 

Sorry for the strange syntax, this is from an early version of my program written in autohotkey.

The language I'm using now is C#, but I assume the solution uses a .dll, so that's not as important. Feel free to suggest or change my tags.

1 Answers

Answers 1

  1. You can try to make inactive (WS_DISABLED - use GetWindowStyle and SetWindowStyle) the main window of destination application (the Window that contains menu).

  2. You can try to find which functions are used by application and rewrite them in local copy of application with VirtualProtect and injecting assembler (dangerous if you have no knowledge about virtualization of memory). Check the application use GetKeyState or GetAsyncKeyState (it will be visible after opening the application in text editor).

  3. You can try: HMENU hMenu=GetMenu(applicationMainWindow); SetMenu(applicationMainWindow,0); // here send your input with SendMessageW instead of PostMessageW SetMenu(applicationMainWindow,hMenu);

Each program can use various methods to handle user keyboard input. In this case probably is used GetAsyncKeyState or GetKeyState (for Ctrl) if you didn't send it and Ctrl is detected.

If it won't help you, please add code with your PostMessage to your question.

BTW. Instead of destroying GetSystemMenu you can clear appropriate window style and after sending input restore it (if the problem is System Menu - probability near 1%).

Read More

Sunday, September 3, 2017

WPF UIAutomation Getting child control of user control

Leave a Comment

I am a user of NordVPN and using it without any issue. Now for some requirements I need to set some of its properties like protocol (checkbox) and clicking on buttons from some other application.

But that area of the application looks like a custom control and UIAutomation is not able to drill down into it.

Elements inside that custom control does not have any automation id.

So I need to know how we could traverse through user controls in wpf applications like other parts of application window using UIAutomation and White Framework.

What I have tried so far is

  1. using TreeWalker (not able to read all elements)

  2. And try to get the element from its location AutomationElement.FromPoint(), but it gives the whole custom control (determine from its bounds) again on which I can't traverse yet.

Any suggestion on how could I drill into custom control from UIAutomation.

enter image description here

For the record, snoop can read the elements but VisualUIAVerify.exe is not.

1 Answers

Answers 1

As expected, the absence of automation-ids have resulted in the controls to be not visible in UI Automation tree APIs.

In order to work around that, and knowing that they are visible in Snoop application - you can use the underlying logic (that Snoop uses) to programmitically automate these controls.

Steps

  1. Download the binaries for SnoopUI, and add them to your project. Make sure to keep the compile option as 'None' and are copied to output directory.

    enter image description here

  2. Next step would be add a helper method, that uses these binaries to inject your dll with automation logic into target application (which is NordVPN) in this case. Once the dll is injected into target process, the ManagedInjector also invokes the method that is sent as parameter.

    public class Helper {     public static void Inject(IntPtr windowHandle, Assembly assembly, string className, string methodName)     {         var location = Assembly.GetEntryAssembly().Location;         var directory = Path.GetDirectoryName(location);         var file = Path.Combine(directory, "HelperDlls", "ManagedInjectorLauncher" + "64-4.0" + ".exe");          Debug.WriteLine(file + " " + windowHandle + " \"" + assembly.Location + "\" \"" + className + "\" \"" + methodName + "\"");         Process.Start(file, windowHandle + " \"" + assembly.Location + "\" \"" + className + "\" \"" + methodName + "\"");     } } 
  3. After the automation dll is injected in the application, the access to Visual Tree is pretty simple using Dispatcher and PresentationSources.

    public class Setup {     public static bool Start()     {         Dispatcher dispatcher;         if (Application.Current == null)             dispatcher = Dispatcher.CurrentDispatcher;         else             dispatcher = Application.Current.Dispatcher;          dispatcher.Invoke(AutomateApp);         return true;     }      public static void AutomateApp()     {         Window root = null;         foreach (PresentationSource presentationSource in PresentationSource.CurrentSources)         {             root = presentationSource.RootVisual as Window;              if (root == null)                 continue;              if ("NordVPN ".Equals(root.Title))                 break;         } 
  4. Getting access to VisualTree is easy, but identifying the controls is not that simple, as there are no automation-id(s), or name(s) that can uniquely identify these controls. But fortunately, as they are using MVVM, it is possible to identify them using the binding(s) attached with them.

    public static T GetChildWithPath<T>(this DependencyObject depObj, DependencyProperty property = null, string pathName = null) where T : DependencyObject {     T toReturn = null;      for (int i = 0; i < VisualTreeHelper.GetChildrenCount(depObj); i++)     {         var child = VisualTreeHelper.GetChild(depObj, i);         bool pathNameMatch = (child is T) && child.IsPathNameMatch<T>(property, pathName);         if (pathNameMatch)         {             toReturn = child as T;             break;         }         else             toReturn = GetChildWithPath<T>(child, property, pathName);          if (toReturn != null)             break;     }     return toReturn; } 
  5. Once you have access to the controls, it is now possible to either manipulate their properties directly, or access their corresponding automation peers, and providers to automate these controls.

    var checkBoxNames = new[] {     "CyberSec", "AutomaticUpdates", "AutoConnect",     "StartOnStartup", "KillSwitch", "ShowNotifications",     "StartMinimized", "ShowServerList", "ShowMap",     "UseCustomDns", "ObfuscatedServersOnly" };  foreach(var path in checkBoxNames) {     var chkBox = settingsView.GetChildWithPath<CheckBox>(CheckBox.IsCheckedProperty, path);     if(chkBox != null && chkBox.IsEnabled)         chkBox.SimulateClick(); } 

A complete working sample has been uploaded at Github repository.

enter image description here

Read More

Tuesday, March 28, 2017

AutomationElement does not retrieve unvisible elements

Leave a Comment

I am trying to get all elements in my Skype program (including all chat tabs), but I get only the visible items.

This it the code:

var window = AutomationElement.RootElement.FindFirst(TreeScope.Subtree,                 new PropertyCondition(AutomationElement.ClassNameProperty, "tSkMainForm"));  if (window != null) {     var items = window.FindAll(TreeScope.Subtree, Condition.TrueCondition);     //DO SOME CODE... } 

The items property does not contain all unvisible items (for example, inner details of chat with someone, let's say, Dan). But if the chat with Dan is opened on my Skype, then the items property would contain also inner details of this chat with Dan. I want the items property to have the chat inner details even if the tab is not opened in my skype.

Why my code does not retrieve all data? How could I get all data (including all chat tabs even when they are not opened)?

1 Answers

Answers 1

iterating through all GridControl rows, use the IScrollProvider interface implementation of GridControlAutomationPeer

private void Button_Click_1(object sender, RoutedEventArgs e) {             var p = Process.GetProcessesByName(ProcName).FirstOrDefault(x => x != null);             if (p == null) {                 Console.WriteLine("proccess: {0} was not found", ProcName); return;             }             var root = AutomationElement.RootElement.FindChildByProcessId(p.Id);             AutomationElement devexGridAutomationElement = root.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.AutomationIdProperty, DevexGridAutomationId));             if (devexGridAutomationElement == null) {                 Console.WriteLine("No AutomationElement was found with id: {0}", DevexGridAutomationId);                 return;             }              var cond = new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.DataItem);             var devexGridItems = devexGridAutomationElement.FindAll(TreeScope.Descendants, cond);             GridPattern gridPat = devexGridAutomationElement.GetCurrentPattern(GridPattern.Pattern) as GridPattern;             Console.WriteLine("number of elements in the grid: {0}", gridPat.Current.RowCount);         } 
Read More

Monday, July 4, 2016

UIA can't get compareendpoints to work between text selection and documentrange in internet explorer

Leave a Comment

Main problem: Can't get CompareEndpoints to give any value other than "1" when comparing the textrange of the selected text with the documentrange on the current site (displayed in IE).

//Initialize range variables IUIAutomationTextRange* documentRange = NULL; IUIAutomationTextRange* selectionRange = NULL; IUIAutomationTextRangeArray* selectionRangeArray = NULL;  //Get entire text document range m_pTextPattern->get_DocumentRange(&documentRange);  //Get selection range m_pTextPattern->GetSelection(&selectionRangeArray); selectionRangeArray->GetElement(0, &selectionRange); 

The ranges are valid, and the selected text is inside the document range. When we try to get the number of moves/characters the selected text is from the start of the document/site-start, then we only get return value of 1.

selectionRange->CompareEndpoints(    TextPatternRangeEndpoint::TextPatternRangeEndpoint_Start,     documentRange,     TextPatternRangeEndpoint::TextPatternRangeEndpoint_Start,    &rv); 

Ex. the site: http://www.cplusplus.com/reference/string/string/

We retrieve the textpattern from the node with name "string - C++ Reference". Then we get the document range of the entire document "documentRange" and select some text with the mouse and saves that range to selectionRange ex. "objects that represent" (selection of text from site... line 3 under std::string).

We have tried the same for a notepad window, where compareendpoints returned a valid/correct distance between the points textranges.

Example:

if (SUCCEEDED(hr))     {         IUIAutomationTextRange* documentRange = NULL;         IUIAutomationTextRangeArray* selectionRangeArray = NULL;         IUIAutomationTextRange* selectionRange = NULL;         hr = E_FAIL;          hr = m_pTextPattern->get_DocumentRange(&documentRange);         if (SUCCEEDED(hr) && documentRange != NULL)         {             hr = m_pTextPattern->GetSelection(&selectionRangeArray);             if (SUCCEEDED(hr) && selectionRangeArray != NULL)             {                 int length;                 hr = selectionRangeArray->get_Length(&length);                 if (SUCCEEDED(hr) && length > 0)                 {                     hr = selectionRangeArray->GetElement(0, &selectionRange);                     if (SUCCEEDED(hr) && selectionRange != NULL)                     {                         hr =  selectionRange->CompareEndpoints(TextPatternRangeEndpoint::TextPatternRangeEndpoint_Start,                              documentRange, TextPatternRangeEndpoint::TextPatternRangeEndpoint_Start, &rv);                         wprintf(L"getSelectionStart rv: %d\n", rv);                              }                 }             }         }         if (documentRange != NULL)         {             documentRange->Release();             documentRange = NULL;         }         if (selectionRangeArray != NULL)         {             selectionRangeArray->Release();             selectionRangeArray = NULL;         }         if (selectionRange != NULL)         {             selectionRange->Release();             selectionRange = NULL;         }      } } 

1 Answers

Answers 1

The docs state that a negative, positive, or zero value is returned. It does not return a distance necessarily.

Read More

Thursday, March 10, 2016

Considerations when installing a desktop WPF app with uiAccess=True

Leave a Comment

Background:

I have a requirement to create a dimming effect on another monitor. I think I solved it by using a WPF Window that takes up the entire screen dimensions with Topmost and AllowsTransparency = True. It has an inner black glow effect and has the style WS_EX_TRANSPARENT | WS_EX_TOOLWINDOW applied to it (among other things) to allow users to click through to the apps behind it.

I monitor for EVENT_OBJECT_REORDER events in Windows and call SetWindowPos to force the Topmost state above other Topmost windows. It seems to work well so far in my proof of concept testing.

The problem I found was this dimming (window) would cover the task bar, but not if I click the Start Menu. I'm currently testing with Windows 10. If I click the Start Menu, it causes the Start Menu and Taskbar to appear above the dimming (window). I wanted everything to remain dim, always.

I solved this issue by setting uiAccess=true in the app manifest, generating a self-signed cert, and copying the exe over to "c:\program files*". This allows me to force a Topmost state for my window, even above the Start Menu.

My questions:

  • Is there a way to position a window over the Start Menu without uiAccess? Or even another way to force dimness to a screen without using a window (but not dependent on monitor drivers or hardware capabilities)?

  • If not, what considerations do I need to keep in mind when distributing a WPF app (via a WiX setup project or something similar) that is to bypass UIPI restrictions with uiAccess=True? Can I simply install my self signed cert during the setup process? Will the user run into any additional hurdles? Will I, as a developer, run into any additional hurdles while building this (aside from what I've already mentioned)?

Thank you!

2 Answers

Answers 1

I monitor for EVENT_OBJECT_REORDER events

You are using SetWinEventHook(). This scenario fails the classic "what if two programs do this" bracket. Raymond Chen discussed this pretty well in this blog post, giving your approach a dedicated post.

This is a lot more common than you might assume. Every Windows machine has a program that does this for example, run Osk.exe, the on-screen keyboard program. Interesting experiment, I predict it will flicker badly for a while but assume it will eventually give up. Not actually sure it does, last time I tried this was at Vista time and it wouldn't, please let us know.

Fairly sure you will conclude that this isn't the right way to go about it so uiAccess is moot as well. You needed it here to bypass UIPI and make SetWindowPos() work. An aspect of UAC that blocks attempts by a program to hijack an elevated program's capabilities. Covering the Start window qualifies as a DOS attack. Bigger problem here is that your self-signed certificate isn't going to work, you'll have to buy a real one. Sets you back several hundred dollars every ~7 years.

Controlling monitor brightness with software isn't that easy to do correctly. Everybody reaches for SetDeviceGammaRamp() and that is what you should do as well. The MSDN docs will give you plenty of FUD but afaik every mainstream video adapter driver implements it. It was popular in games. One unavoidable limitation is that it is only active for the desktop in which your program runs. So not for the secure desktop (screen saver and Ctrl+Alt+Del) and not for other login sessions unless they start your program as well.

WMI is too flaky to consider. Not so sure why it fails so often, I assume it has something to do with the often less-than-stellar I2C interconnect between the video adapter and the monitor. Or laptops that want to control brightness with an Fn keystroke, that feature always wins. Or the Windows feature that automatically adjusts brightness based on ambient light, invariably the more desirable way to do this and a hard act to follow.

Most common outcome is likely to be a shrug at your program and a curse of the user at the clumsy monitor controls. But he'll fiddle with it and figure it out. Sorry.

Answers 2

This won't answer anything about uiAccess=true, but...

Dimming the Screen

As an alternative way to dim the screen, you could try using SetDeviceGammaRamp to dim all screens at once (if that's desired).

For example, take the following helper class:

/// <summary> Allows changing the gamma of the displays. </summary> public static class GammaChanger {   /// <summary>   ///  Retrieves the current gamma ramp data so that it can be restored later.   /// </summary>   /// <param name="gamma"> [out] The current gamma. </param>   /// <returns> true if it succeeds, false if it fails. </returns>   public static bool GetCurrentGamma(out GammaRampRgbData gamma)   {     gamma = GammaRampRgbData.Create();     return GetDeviceGammaRamp(GetDC(IntPtr.Zero), ref gamma);   }    public static bool SetGamma(ref GammaRampRgbData gamma)   {     // Now set the value.     return SetDeviceGammaRamp(GetDC(IntPtr.Zero), ref gamma);   }    public static bool SetBrightness(int gamma)   {     GammaRampRgbData data = new GammaRampRgbData                             {                               Red = new ushort[256],                               Green = new ushort[256],                               Blue = new ushort[256]                             };      int wBrightness = gamma; // reduce the brightness     for (int ik = 0; ik < 256; ik++)     {       int iArrayValue = ik * (wBrightness + 128);       if (iArrayValue > 0xffff)       {         iArrayValue = 0xffff;       }       data.Red[ik] = (ushort)iArrayValue;       data.Green[ik] = (ushort)iArrayValue;       data.Blue[ik] = (ushort)iArrayValue;     }      return SetGamma(ref data);   }    [DllImport("gdi32.dll")]   private static extern bool SetDeviceGammaRamp(IntPtr hdc, ref GammaRampRgbData gammaRgbArray);    [DllImport("gdi32.dll")]   private static extern bool GetDeviceGammaRamp(IntPtr hdc, ref GammaRampRgbData gammaRgbArray);    [DllImport("user32.dll")]   private static extern IntPtr GetDC(IntPtr hWnd);    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]   public struct GammaRampRgbData   {     [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)]     public UInt16[] Red;      [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)]     public UInt16[] Green;      [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)]     public UInt16[] Blue;      /// <summary> Creates a new, initialized GammaRampRgbData object. </summary>     /// <returns> A GammaRampRgbData. </returns>     public static GammaRampRgbData Create()     {       return new GammaRampRgbData              {                Red = new ushort[256],                Green = new ushort[256],                Blue = new ushort[256]              };     }   } } 

Combined with the following in a static void Main(), and the program will change the brightness until the user exits the application:

GammaChanger.GammaRampRgbData originalGamma; bool success = GammaChanger.GetCurrentGamma(out originalGamma); Console.WriteLine($"Originally: {success}");  success = GammaChanger.SetBrightness(44); Console.WriteLine($"Setting: {success}");  Console.ReadLine();  success = GammaChanger.SetGamma(ref originalGamma); Console.WriteLine($"Restoring: {success}");  Console.ReadLine(); 

Do note however, that this is applying a global solution to a local problem

If you do go this route, I'd suggest really making sure that you're restoring the user's gamma before exiting, otherwise they'll be left with a less than steller experience that your app crashed and the screen is no permanently dimmed.

Sources:

Read More