Add support for shadow DOM #604 (#606)

Implement and document shadow DOM functions (#604)

CHANGELOG.adoc

@@ -10,6 +10,7 @@ A release with an intentional breaking changes is marked with:
 * https://github.com/clj-commons/etaoin/pull/552[#552]: Add support for wide characters to input fill functions
 (https://github.com/tupini07[@tupini07])
 * https://github.com/clj-commons/etaoin/issues/566[#566]: Recognize `:driver-log-level` for Edge
+* https://github.com/clj-commons/etaoin/issues/604[#604]: Add support for shadow DOM
 * bump all deps to current versions
 * tests
 ** https://github.com/clj-commons/etaoin/issues/572[#572]: stop using chrome `--no-sandbox` option, it has become problematic on Windows (and we did not need it anyway)

doc/01-user-guide.adoc

@@ -845,6 +845,125 @@ The following query will find a vector of `div` tags, then return a set of all `
 ;; => ("a1" "a2" "a3" "a4" "a5" "a6" "a7" "a8" "a9")
 ----
 
+[#shadow-dom]
+=== Querying the Shadow DOM
+
+The shadow DOM provides a way to attach another DOM tree to a specified element in the normal DOM and have the internals of that tree hidden from JavaScript and CSS on the same page.
+When the browser renders the DOM, the elements from the shadow DOM appear at the location where the tree is rooted in the normal DOM.
+This provides a level of encapsulation, allowing "components" in the shadow DOM to be styled differently than the rest of the page and preventing conflicts between the normal page CSS and the component CSS.
+The shadow DOM is also hidden from normal Web Driver queries (`query`) and thus requires a separate set of API calls to query it. For more details about the shadow DOM, see this article at https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_shadow_DOM#shadow_dom_and_custom_elements[Mozilla Developer Network (MDN)].
+
+There are a few terms that are important to understand when dealing with the Shadow DOM.
+The "shadow root host" is the element in the standard DOM to which a shadow root is attached as a property.
+The "shadow root" is the top of the shadow DOM tree rooted at the shadow root host.
+
+The following examples use this HTML fragment in the User Guide sample HTML that has a bit of shadow DOM in it.
+
+[source,html]
+----
+<span id="not-in-shadow">I'm not in the shadow DOM</span>
+<div id="shadow-root-host">
+    <template shadowrootmode="open">
+        <span id="in-shadow">I'm in the shadow DOM</span>
+        <span id="also-in-shadow">I'm also in the shadow DOM</span>
+    </template>
+</div>
+----
+
+Everthing in the `template` element is part of the shadow DOM.
+The `div` with the `id` of `shadow-root-host` is, as the ID suggests, the shadow root host element.
+
+Given this HTML, you can run a standard `query` to find the shadow root host and then use `get-element-property-el` to return to the `"shadowRoot"` property.
+Note that the element IDs returned in the following examples will be unique to the specific Etaoin driver and driver session and you will not see the same IDs.
+
+[source,clojure]
+----
+(e/query driver {:id "shadow-root-host"})
+;; an element ID similar to (but not the same as)
+;; "78344155-7a53-46fb-a46e-e864210e501d"
+
+(e/get-element-property-el driver (e/query driver {:id "shadow-root-host"}) "shadowRoot")
+;; something similar to
+;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}
+
+(e/get-element-property driver {:id "shadow-root-host"} "shadowRoot")
+;; something similar to
+;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}
+----
+
+If you go this route, you're going to have to pick apart the return
+values.
+The element-id of the shadow root is the string value of the first map key.
+
+You can get the shadow root element ID more directly using Etaoin's `get-element-shadow-root` API.
+The query parameter looks for a matching element in the standard DOM and returns its shadow root property.
+
+[source,clojure]
+----
+(e/get-element-shadow-root driver {:id "shadow-root-host"})
+;; something similar to
+;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"
+----
+
+If you already have the shadow root host element, you can return its corresponding shadow root element ID using `get-element-shadow-root-el`.
+
+[source,clojure]
+----
+(def host (e/query driver {:id "shadow-root-host"}))
+(e/get-element-shadow-root-el driver host)
+;; something similar to
+;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"
+----
+
+You can test whether an element is a shadow root host using `has-shadow-root?` and `has-shadow-root-el?`.
+
+[source,clojure]
+----
+(e/has-shadow-root? driver {:id "shadow-root-host"})
+;; => true
+(e/has-shadow-root-el? driver host)
+;; => true
+(e/has-shadow-root? driver {:id "not-in-shadow"})
+;; => false
+----
+
+Now that you know how to retrieve the shadow root, you can query elements in the shadow DOM using `query-shadow-root`, `query-all-shadow-root`, `query-shadow-root-el`, and `query-all-shadow-root-el`.
+
+For `query-shadow-root` and `query-all-shadow-root`, the `q` parameter specifies a query of the _normal_ DOM to find the shadow root host.
+If the host is identified, the `shadow-q` parameter is a query that is executed within the shadow DOM rooted at the shadow root host.
+
+The `query-shadow-root-el` and `query-all-shadow-root-el` allow you to specify the shadow root host element directly, rather than querying for it.
+
+[source,clojure]
+----
+(def in-shadow (e/query-shadow-root driver {:id "shadow-root-host"} {:css "#in-shadow"}))
+(e/get-element-text-el driver in-shadow)
+;; => "I'm in the shadow DOM"
+
+(->> (e/query-all-shadow-root driver {:id "shadow-root-host"} {:css "span"})
+     (map #(e/get-element-text-el driver %)))
+;; => ("I'm in the shadow DOM" "I'm also in the shadow DOM")
+
+(def shadow-root (e/get-element-shadow-root-el driver host))
+(e/get-element-text-el driver (e/query-shadow-root-el driver shadow-root {:css "#in-shadow"}))
+;; => "I'm in the shadow DOM"
+ 
+(->> (e/query-all-shadow-root-el driver shadow-root {:css "span"})
+     (map #(e/get-element-text-el driver %)))
+;; > ("I'm in the shadow DOM" "I'm also in the shadow DOM")
+----
+
+[#shadow-root-browser-limitations]
+[NOTE]
+====
+In the previous shadow root queries, you should note that we used CSS selectors for the `shadow-q` argument in each case.
+This was done because current browsers do not support XPath, which is what the Etaoin map syntax is typically translated into under the hood.
+While it is expected that browsers will support XPath queries of the shadow DOM in the future, it is unclear when this support might appear.
+For now, use CSS.
+
+For more information, see the https://wpt.fyi/results/webdriver/tests/classic/find_element_from_shadow_root/find.py?label=experimental&label=master&aligned[Web Platforms Test Dashobard].
+====
+
 === Interacting with Queried Elements
 
 To interact with elements found via a `query` or `query-all` function call you have to pass the query result to either `click-el` or `fill-el` (note the `-el` suffix):

doc/user-guide-sample.html

@@ -140,6 +140,15 @@ <h2>Query Tree Example</h2>
     </div>
   </div>
 
+  <h2>Shadow DOM Example</h2>
+  <span id="not-in-shadow">I'm not in the shadow DOM</span>
+  <div id="shadow-root-host">
+      <template shadowrootmode="open">
+          <span id="in-shadow">I'm in the shadow DOM</span>
+          <span id="also-in-shadow">I'm also in the shadow DOM</span>
+      </template>
+  </div>
+
   <h2>Frames</h2>
   <p id="in-main-page">In main page paragraph</p>
   <iframe id="frame1" src="user-guide-sample-frame1.html"></iframe>

env/test/resources/static/test.html

@@ -238,6 +238,15 @@ <h3>Find text with quote</h3>
             </p>
         </div>
 
+        <h3>Shadow DOM</h3>
+        <span id="not-in-shadow">I'm not in the shadow DOM</span>
+        <div id="shadow-root-host">
+            <template shadowrootmode="open">
+                <span id="in-shadow">I'm in the shadow DOM</span>
+                <span id="also-in-shadow">I'm also in the shadow DOM</span>
+            </template>
+        </div>
+
         <h3 id="document-end">Document end</h3>
 
     </body>

src/etaoin/api.clj

@@ -25,6 +25,8 @@
 
   **Querying/Selecting DOM Elements**
   - [[query]] [[query-all]] [[query-tree]]
+  - [[query-shadow-root]] [[query-shadow-root-el]] [[query-all-shadow-root]] [[query-all-shadow-root-el]]
+  - [[has-shadow-root?]] [[has-shadow-root-el?]]
   - [[exists?]] [[absent?]]
   - [[displayed?]] [[displayed-el?]] [[enabled?]] [[enabled-el?]] [[disabled?]] [[invisible?]] [[visible?]]
   - [[child]] [[children]]
@@ -33,6 +35,7 @@
   - [[get-element-property]] [[get-element-property-el]] [[get-element-properties]]
   - [[has-class?]] [[has-class-el?]] [[has-no-class?]]
   - [[get-element-css]] [[get-element-css-el]] [[get-element-csss]]
+  - [[get-element-shadow-root]] [[get-element-shadow-root-el]]
   - [[get-element-text]] [[get-element-text-el]] [[has-text?]]
   - [[get-element-inner-html]] [[get-element-inner-html-el]]
   - [[get-element-value]] [[get-element-value-el]]
@@ -1611,6 +1614,110 @@
   [driver q]
   (get-element-property driver q :value))
 
+(defn get-element-shadow-root-el
+  "Returns the shadow root for the specified element or `nil` if the
+  element does not have a shadow root."
+  [driver el]
+  (-> (get-element-property-el driver el "shadowRoot")
+      first
+      second))
+
+(defn get-element-shadow-root
+  "Returns the shadow root for the first element matching the query, or
+  `nil` if the element does not have a shadow root.
+
+  See [[query]] for more details on `q`."
+  [driver q]
+  (get-element-shadow-root-el driver (query driver q)))
+
+;;; 
+;;; Shadow root queries
+;;; 
+
+(defmulti ^:private find-element-from-shadow-root* dispatch-driver)
+
+(defmethods find-element-from-shadow-root*
+  [:firefox :safari]
+  [driver shadow-root-el locator term]
+  {:pre [(some? shadow-root-el)]}
+  (-> (execute {:driver driver
+                :method :post
+                :path   [:session (:session driver) :shadow shadow-root-el :element]
+                :data   {:using locator :value term}})
+      :value
+      first
+      second))
+
+(defmethod find-element-from-shadow-root* :default
+  [driver shadow-root-el locator term]
+  {:pre [(some? shadow-root-el)]}
+  (-> (execute {:driver driver
+                :method :post
+                :path   [:session (:session driver) :shadow shadow-root-el :element]
+                :data   {:using locator :value term}})
+      :value
+      :ELEMENT))
+
+(defmulti ^:private find-elements-from-shadow-root* dispatch-driver)
+
+(defmethod find-elements-from-shadow-root* :default
+  [driver shadow-root-el locator term]
+  {:pre [(some? shadow-root-el)]}
+  (->> (execute {:driver driver
+                 :method :post
+                 :path   [:session (:session driver) :shadow shadow-root-el :elements]
+                 :data   {:using locator :value term}})
+       :value
+       (mapv (comp second first))))
+
+(defn query-shadow-root-el
+  "Queries the shadow DOM rooted at `shadow-root-el`, looking for the
+  first element specified by `shadow-q`.
+
+  The `shadow-q` parameter is similar to the `q` parameter of
+  the [[query]] function, but some drivers may limit it to specific
+  formats (e.g., CSS). See [this note](/doc/01-user-guide.adoc#shadow-root-browser-limitations) for more information."
+  [driver shadow-root-el shadow-q]
+  (let [[loc term] (query/expand driver shadow-q)]
+    (find-element-from-shadow-root* driver shadow-root-el loc term)))
+
+(defn query-all-shadow-root-el
+  "Queries the shadow DOM rooted at `shadow-root-el`, looking for all
+  elements specified by `shadow-q`.
+
+  The `shadow-q` parameter is similar to the `q` parameter of
+  the [[query]] function, but some drivers may limit it to specific
+  formats (e.g., CSS). See [this note](/doc/01-user-guide.adoc#shadow-root-browser-limitations) for more information."
+  [driver shadow-root-el shadow-q]
+  (let [[loc term] (query/expand driver shadow-q)]
+    (find-elements-from-shadow-root* driver shadow-root-el loc term)))
+
+(defn query-shadow-root
+  "First, conducts a standard search (as if by [[query]]) for an element
+  with a shadow root. Then, from that shadow root element, conducts a
+  search of the shadow DOM for the first element matching `shadow-q`.
+
+  For details on `q`, see [[query]].
+
+  The `shadow-q` parameter is similar to the `q` parameter of
+  the [[query]] function, but some drivers may limit it to specific
+  formats (e.g., CSS). See [this note](/doc/01-user-guide.adoc#shadow-root-browser-limitations) for more information."
+  [driver q shadow-q]
+  (query-shadow-root-el driver (get-element-shadow-root driver q) shadow-q))
+
+(defn query-all-shadow-root
+  "First, conducts a standard search (as if by [[query]]) for an element
+  with a shadow root. Then, from that shadow root element, conducts a
+  search of the shadow DOM for all elements matching `shadow-q`.
+
+  For details on `q`, see [[query]].
+
+  The `shadow-q` parameter is similar to the `q` parameter of
+  the [[query]] function, but some drivers may limit it to specific
+  formats (e.g., CSS). See [this note](/doc/01-user-guide.adoc#shadow-root-browser-limitations) for more information."
+  [driver q shadow-q]
+  (query-all-shadow-root-el driver (get-element-shadow-root driver q) shadow-q))
+
 ;;
 ;; cookies
 ;;
@@ -2431,6 +2538,17 @@
        :arglists '([driver])}
   has-no-alert? (complement has-alert?))
 
+(defn has-shadow-root-el?
+  "Returns `true` if the specified element has a shadow root or `false` otherwise."
+  [driver el]
+  (boolean (get-element-shadow-root-el driver el)))
+
+(defn has-shadow-root?
+  "Returns `true` if the first element matching the query has a shadow
+  root or `false` otherwise."
+  [driver q]
+  (boolean (get-element-shadow-root driver q)))
+
 ;;
 ;; wait functions
 ;;

test/etaoin/api_test.clj

@@ -799,6 +799,45 @@
         (e/wait 1)
         (is (str/ends-with? (e/get-url *driver*) "?login=1&password=2&message=3"))))))
 
+(deftest test-shadow-dom
+  (testing "basic functional sanity"
+    ;; Validate that the test DOM is as we would expect
+    (is (e/has-text? *driver* {:id "not-in-shadow"} "I'm not in the shadow DOM"))
+    (is (not (e/has-text? *driver* {:id "in-shadow"} "I'm in the shadow DOM"))))
+  (testing "getting the shadow root for an element"
+    (is (some? (e/get-element-shadow-root *driver* {:id "shadow-root-host"})))
+    (is (some? (e/get-element-shadow-root-el *driver*
+                                             (e/query *driver* {:id "shadow-root-host"})))))
+  (testing "whether an element has a shadow root"
+    (is (e/has-shadow-root? *driver* {:id "shadow-root-host"}))
+    (is (e/has-shadow-root-el? *driver* (e/query *driver* {:id "shadow-root-host"}))))
+  (let [shadow-root (e/get-element-shadow-root *driver* {:id "shadow-root-host"})]
+    (testing "querying the shadow root element for a single element"
+      (is (= "I'm in the shadow DOM"
+             (->> (e/query-shadow-root-el *driver*
+                                          shadow-root
+                                          {:css "#in-shadow"})
+                  (e/get-element-text-el *driver*))))
+      (is (= "I'm also in the shadow DOM"
+             (->> (e/query-shadow-root-el *driver*
+                                          shadow-root
+                                          {:css "#also-in-shadow"})
+                  (e/get-element-text-el *driver*)))))
+    (testing "querying the shadow root element for multiple elements"
+      (is (= ["I'm in the shadow DOM" "I'm also in the shadow DOM"]
+             (->> (e/query-all-shadow-root-el *driver*
+                                              shadow-root
+                                              {:css "span"})
+                  (mapv #(e/get-element-text-el *driver* %)))))))
+  (testing "querying the shadow root element"
+    (is (= "I'm in the shadow DOM"
+           (->> (e/query-shadow-root *driver* {:id "shadow-root-host"} {:css "#in-shadow"})
+                (e/get-element-text-el *driver*)))))
+  (testing "querying the shadow root element for multiple elements"
+    (is (= ["I'm in the shadow DOM" "I'm also in the shadow DOM"]
+           (->> (e/query-all-shadow-root *driver* {:id "shadow-root-host"} {:css "span"})
+                (mapv #(e/get-element-text-el *driver* %)))))))
+
 (comment
   ;; start test server
   (def test-server (p/process {:out :inherit :err :inherit} "bb test-server --port" 9993))