From f64e08b9c55d326be9d3c2ef681b79dcc7a74899 Mon Sep 17 00:00:00 2001 From: Jeff Matthews Date: Fri, 16 Sep 2022 11:12:02 -0500 Subject: [PATCH 1/8] Removed obsolete docs/ directory --- docs/backward-incompatible-changes.md | 163 -- docs/best-practices.md | 217 -- docs/commands/codeception.md | 93 - docs/commands/mftf.md | 683 ----- docs/configuration.md | 446 --- docs/configure-2fa.md | 53 - docs/credentials.md | 290 -- docs/custom-helpers.md | 129 - docs/data.md | 345 --- docs/debugging.md | 37 - docs/extending.md | 342 --- docs/getting-started.md | 368 --- docs/guides/action-groups.md | 82 - docs/guides/cicd.md | 114 - docs/guides/git-vs-composer-install.md | 83 - docs/guides/selectors.md | 346 --- docs/guides/test-isolation.md | 119 - docs/guides/test-modularity.md | 88 - docs/guides/using-suites.md | 101 - docs/img/action-groups-dia.svg | 516 ---- .../catalogCategoryRepository-operations.png | Bin 15654 -> 0 bytes docs/img/data-dia.svg | 489 ---- docs/img/metadata-dia.svg | 1050 ------- docs/img/mftf-extending-precedence.png | Bin 127088 -> 0 bytes docs/img/mftf-fork.gif | Bin 387415 -> 0 bytes docs/img/page-dia.svg | 290 -- docs/img/section-dia.svg | 296 -- docs/img/test-dia.svg | 1228 -------- docs/interactive-pause.md | 74 - docs/introduction.md | 163 -- docs/merge_points/extend-action-groups.md | 102 - docs/merge_points/extend-data.md | 70 - docs/merge_points/extend-tests.md | 145 - docs/merge_points/introduction.md | 39 - docs/merge_points/merge-action-groups.md | 78 - docs/merge_points/merge-data.md | 56 - docs/merge_points/merge-pages.md | 50 - docs/merge_points/merge-sections.md | 46 - docs/merge_points/merge-tests.md | 102 - docs/merging.md | 569 ---- docs/metadata.md | 587 ---- docs/mftf-tests-packaging.md | 58 - docs/page.md | 174 -- docs/reporting.md | 313 -- docs/section.md | 154 - docs/section/locator-functions.md | 45 - docs/section/parameterized-selectors.md | 155 - docs/selectors.md | 35 - docs/suite.md | 295 -- docs/test-prep.md | 407 --- docs/test.md | 142 - docs/test/action-groups.md | 306 -- docs/test/actions.md | 2571 ----------------- docs/test/annotations.md | 243 -- docs/test/assertions.md | 704 ----- docs/tips-tricks.md | 423 --- docs/troubleshooting.md | 65 - docs/update.md | 53 - docs/versioning.md | 71 - 59 files changed, 16263 deletions(-) delete mode 100644 docs/backward-incompatible-changes.md delete mode 100644 docs/best-practices.md delete mode 100644 docs/commands/codeception.md delete mode 100644 docs/commands/mftf.md delete mode 100644 docs/configuration.md delete mode 100644 docs/configure-2fa.md delete mode 100644 docs/credentials.md delete mode 100644 docs/custom-helpers.md delete mode 100644 docs/data.md delete mode 100644 docs/debugging.md delete mode 100644 docs/extending.md delete mode 100644 docs/getting-started.md delete mode 100644 docs/guides/action-groups.md delete mode 100644 docs/guides/cicd.md delete mode 100644 docs/guides/git-vs-composer-install.md delete mode 100644 docs/guides/selectors.md delete mode 100644 docs/guides/test-isolation.md delete mode 100644 docs/guides/test-modularity.md delete mode 100644 docs/guides/using-suites.md delete mode 100644 docs/img/action-groups-dia.svg delete mode 100644 docs/img/catalogCategoryRepository-operations.png delete mode 100644 docs/img/data-dia.svg delete mode 100644 docs/img/metadata-dia.svg delete mode 100644 docs/img/mftf-extending-precedence.png delete mode 100644 docs/img/mftf-fork.gif delete mode 100644 docs/img/page-dia.svg delete mode 100644 docs/img/section-dia.svg delete mode 100644 docs/img/test-dia.svg delete mode 100644 docs/interactive-pause.md delete mode 100644 docs/introduction.md delete mode 100644 docs/merge_points/extend-action-groups.md delete mode 100644 docs/merge_points/extend-data.md delete mode 100644 docs/merge_points/extend-tests.md delete mode 100644 docs/merge_points/introduction.md delete mode 100644 docs/merge_points/merge-action-groups.md delete mode 100644 docs/merge_points/merge-data.md delete mode 100644 docs/merge_points/merge-pages.md delete mode 100644 docs/merge_points/merge-sections.md delete mode 100644 docs/merge_points/merge-tests.md delete mode 100644 docs/merging.md delete mode 100644 docs/metadata.md delete mode 100644 docs/mftf-tests-packaging.md delete mode 100644 docs/page.md delete mode 100644 docs/reporting.md delete mode 100644 docs/section.md delete mode 100644 docs/section/locator-functions.md delete mode 100644 docs/section/parameterized-selectors.md delete mode 100644 docs/selectors.md delete mode 100644 docs/suite.md delete mode 100644 docs/test-prep.md delete mode 100644 docs/test.md delete mode 100644 docs/test/action-groups.md delete mode 100644 docs/test/actions.md delete mode 100644 docs/test/annotations.md delete mode 100644 docs/test/assertions.md delete mode 100644 docs/tips-tricks.md delete mode 100644 docs/troubleshooting.md delete mode 100644 docs/update.md delete mode 100644 docs/versioning.md diff --git a/docs/backward-incompatible-changes.md b/docs/backward-incompatible-changes.md deleted file mode 100644 index 5ddf1c9cb..000000000 --- a/docs/backward-incompatible-changes.md +++ /dev/null @@ -1,163 +0,0 @@ -# MFTF 3.0.0 backward incompatible changes - -This page highlights backward incompatible changes between releases that have a major impact and require detailed explanation and special instructions to ensure third-party tests continue working with Magento core tests. - -## Minimum supported PHP version changes - -We changed the minimum PHP version requirement from 7.0 to 7.3. Because of the PHP version requirement change, this MFTF version only supports Magento 2.4 or later. - -## Folder structure changes - -We removed support to read test modules from the deprecated path `dev/tests/acceptance/tests/functional/Magento/FunctionalTest`. If there are test modules in this path, they should be moved to `dev/tests/acceptance/tests/functional/Magento`. - -## XSD schema changes - -- Files under test modules `ActionGroup`, `Page`, `Section`, `Test` and `Suite` only support a single entity per file. -- The `file` attribute from `` has been removed from the suite schema. `` is no longer supported in suites. -- Metadata filename format changed to ***`*Meta.xml`***. -- Only nested assertion syntax will be supported. See the [assertions page](test/assertions.md) for details. Here is an example of the nested assertion syntax: - ```xml - - $billingAddressOrderPage - $shippingAddressOrderPage - - ``` - -### Upgrading tests to the new schema - -The following table lists the upgrade scripts that are available to upgrade tests to the new schema. - -| Script name | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------------| -|`splitMultipleEntitiesFiles`| Splits files that have multiple entities into multiple files with one entity per file. | -|`upgradeAssertionSchema`| Updates assert actions that uses the old assertion syntax into the new nested syntax.| -|`renameMetadataFiles`| Renames Metadata filenames to `*Meta.xml`.| -|`removeModuleFileInSuiteFiles`| Removes occurrences of `` from all ``s.| -|`removeUnusedArguments`| Removes unused arguments from action groups.| -|`upgradeTestSchema`| Replaces relative schema paths to URN in test files.| - -To run the upgrade tests: - -1. Run `bin/mftf reset --hard` to remove old generated configurations. -1. Run `bin/mftf build:project` to generate new configurations. -1. Run `bin/mftf upgrade:tests`. [See command page for details](commands/mftf.md#upgradetests). -1. Lastly, try to generate all tests. Tests should all be generated as a result of the upgrades. If not, the most likely issue will be a changed XML schema. Check error messaging and search your codebase for the attributes listed. - -## MFTF commands - -`--debug` option `NONE` removed for strict schema validation. Ensure there are no schema validation errors in test modules before running MFTF commands. - -## MFTF actions - -### `executeInSelenium` and `performOn` removed - -**Action**: Deprecated actions `executeInSelenium` and `performOn` are removed in favor of new action `helper`. - -**Reason**: `executeInSelenium` and `performOn` allowed custom PHP code to be written inline inside of XML files which was difficult to maintain, troubleshoot, and modify. - -**Details**: - -The `helper` allows test writers to solve advanced requirements beyond what MFTF offers out of the box. See [custom-helpers](custom-helpers.md) for more information on usage. - -Here is an example of using `helper` in place of `executeSelenium` to achieve same workflow. - -Old usage: - -```xml - -``` - -New usage: - -```xml - - //div[contains(@class, 'inline-wysiwyg')]//h2 - {{TinyMCEPartialHeadingSelection.startX}} - {{TinyMCEPartialHeadingSelection.startY}} - {{TinyMCEPartialHeadingSelection.endX}} - {{TinyMCEPartialHeadingSelection.endY}} - -``` - -### `pauseExecution` removed - -**Action**: `pauseExecution` is removed in favor of `pause`. - -**Reason**: `[WebDriver]pauseExecution` is removed in Codeception 3 in favor of `I->pause()`. - -**Details**: - -See the [actions page for details](test/actions.md#pause). Here is a usage example: - -```xml - -``` - -### Removed assert actions - -**Action**: Assert actions `assertInternalType`, `assertNotInternalType` and `assertArraySubset` are removed. - -**Reason**: PHPUnit 9 has dropped support for these assertions. - -### Updated assert actions - -**Action**: The `delta` attribute has been removed from `assertEquals` and `assertNotEquals`. Instead, new assert actions have been introduced: - - - `assertEqualsWithDelta` - - `assertNotEqualsWithDelta` - - `assertEqualsCanonicalizing` - - `assertNotEqualsCanonicalizing` - - `assertEqualsIgnoringCase` - - `assertNotEqualsIgnoringCase` - -**Reason**: PHPUnit 9 has dropped support for optional parameters for `assertEquals` and `assertNotEquals` and has introduced these new assertions. - -**Details**: - -Usage of `assertEquals` or `assertNotEquals` with a specified `delta`, should be replaced with appropriate assertion from the above list. - -### `assertContains` supports only iterable haystacks - -**Action**: `assertContains` and `assertNotContains` now only supports iterable haystacks. These assert actions have been added to work with string haystacks: - -- `assertStringContainsString` -- `assertStringNotContainsString` -- `assertStringContainsStringIgnoringCase` -- `assertStringNotContainsStringIgnoringCase` - -**Reason**: With PHPUnit 9, `assertContains` and `assertNotContains` only allows iterable haystacks. New assertions have been introduced to support string haystacks. - -**Details**: - -Usages of `assertContains` and `assertNotContains` with string haystacks should be replaced with appropriate assertion from the above list. - -Usage example for string haystacks: - -```xml - - $grabSimpleProdPrice2 - $110.70 - -``` - -### `formatMoney` removed - -**Action**: `formatMoney` has been removed in favor of `formatCurrency`. - -**Reason**: PHP 7.4 has deprecated use of `formatMoney`. - -**Details**: Format input to specified currency according to the locale specified. - -Usage example: - -```xml - -``` diff --git a/docs/best-practices.md b/docs/best-practices.md deleted file mode 100644 index 1f3a25e02..000000000 --- a/docs/best-practices.md +++ /dev/null @@ -1,217 +0,0 @@ -# Best practices - -Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework. - -## Focus on reusability - -### Use existing Tests and resources - -Magento offers more than **3000** acceptance tests, **2500** [Action group]s, **750** Page declarations with more than **1500** Section definitions. -It is very probable that behaviour you want to test already exists as a Test or Action Group. -Instead of writing everything by yourself - use `extends` attribute to refer to existing element and customize it. - -**Reusable Resources** - -{%raw%} - -* Tests (reusable with `` argument) -* Action Group (reusable with including ``, or extending ``) -* Pages (reusable with reference `{{PageDefinition.url}}`) -* Sections (reusable with reference `{{SectionDefinition.elementDefinition}}`) -* Data Entities (reusable with reference `"` or extending ``) - -{%endraw%} - -
- -Avoid using resources that are marked as **Deprecated**. Usually there is a replacement provided for a deprecated resource. - -
- -### Extract repetitive Actions - -Instead of writing a few of Tests that perform mostly the same actions, you should thing about [Action group] that is a container for repetitive Actions. -If each run needs different data, use `` to inject necessary information. - -We recommend to keep Action Groups having single responsibility, for example `AdminLoginActionGroup`, which expected outcome is being logged in as Administrator when [Action group] is executed. - -## Contribute - -Although the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. -If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute. - -## Action group - -1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed. -2. Provide default values for the arguments that apply to your most common case scenarios. -3. One `` tag is allowed per action group XML file. - -## `actionGroups` vs `extends` - -Use an action group to wrap a set of actions to reuse them multiple times. - -Use an [extension] when a test or action group needs to be repeated with the exception of a few steps. - -### When to use `extends` - -Use `extends` in your new test or action group when at least one of the following conditions is applicable to your case: - -1. You want to keep the original test without any modifications. -2. You want to create a new test that follows the same path as the original test. -3. You want a new action group that behaves similarly to the existing action group, but you do not want to change the functionality of the original action group. - -### When to avoid `extends` - -Do not use `extends` in the following conditions: - -1. You want to change the functionality of the test or action group and do not need to run the original version. -2. You plan to merge the base test or action group. - -The following pattern is used when merging with `extends`: - -1. The original test is merged. -2. The extended test is created from the merged original test. -3. The extended test is merged. - -## Annotation - -1. Use [annotations] in a test. -2. Update your annotations correspondingly when updating tests. - -## Data entity - -1. Keep your testing instance clean. - Remove data after the test if the test required creating any data. - Use a corresponding [``] test step in your [``] block when using a [``] action in a [``] block. -2. Make specific data entries under test to be unique. - Enable data uniqueness where data values are required to be unique in a database by test design. - Use `unique=”suffix”` or `unique=”prefix”` to append or prepend a unique value to the [entity] attribute. - This ensures that tests using the entity can be repeated. -3. Do not modify existing data entity fields or merge additional data fields without complete understanding and verifying the usage of existing data in tests. - Create a new data entity for your test if you are not sure. - -## Naming conventions - -### File names - -Name files according to the following patterns to make searching in future more easy: - - - -#### Test file name - -Format: {_Admin_ or _Storefront_}{Functionality}_Test.xml_, where Functionality briefly describes the testing functionality. - -Example: _StorefrontCreateCustomerTest.xml_. - -#### Action Group file name - -Format: {_Admin_ or _Storefront_}{Action Group Summary}ActionGroup.xml`, where Action Group Summary is a short description of what the action group does. - -Example: _AdminCreateStoreActionGroup.xml_ - -#### Section file name - -Format: {_Admin_ or _Storefront_}{UI Description}_Section.xml_, where UI Description briefly describes the testing UI. - -Example: _AdminNavbarSection.xml_. - -#### Data file name - -Format: {Type}_Data.xml_, where Type represents the entity type. - - - -Example: _ProductData.xml_. - -### Object names - -Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _classProperties_ in PHP. - -#### Upper case - -Use an upper case first letter for: - -* File names. Example: _StorefrontCreateCustomerTest.xml_ -* Test name attributes. Example: `` -* Data entity names. Example: `` -* Page name. Example: `` -* Section name. Example: `
` -* Action group name. Example: `` - -#### Lower case - -Use a lower case first letter for: - -* Data keys. Example: `` -* Element names. Examples: `` -* Step keys. For example: `` - -## Page object - -1. One `` tag is allowed per page XML file. -2. Use [parameterized selectors] for constructing a selector when test-specific or runtime-generated information is needed. -Do not use them for static elements. - - -BAD: - - - - -``` xml - -``` - - - - -GOOD: - - -Define these three elements and reference them by name in the tests. - -``` xml - - - -``` - -## Test - -1. Use actions such as [``], [``], and [``] to wait the exact time required for the test step. - Try to avoid using the [``] action, because it forces the test to wait for the time you specify. You may not need to wait so long to proceed. -1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution. - It also helps you to identify the cause of test failure. -1. Use comments to keep tests readable and maintainable: - * Keep the inline `` and [``] tags up to date. - It helps to inform the reader of what you are testing and to yield a more descriptive Allure report. - * Explain in comments unclear or tricky test steps. -1. Refer to [sections] instead of writing selectors. -1. One `` tag is allowed per test XML file. - -## Test step merging order - -When setting a [merging] order for a test step, do not depend on steps from Magento modules that could be disabled by an application. - -For example, when you write a test step to create a gift card product, set your test step **after** simple product creation and let the MFTF handle the merge order. -Since the configurable product module could be disabled, this approach is more reliable than setting the test step **before** creating a configurable product. - - - -[``]: test/actions.html#before-and-after -[``]: test/actions.html#before-and-after -[``]: test/actions.html#comment -[``]: test/actions.html#createdata -[``]: test/actions.html#deletedata -[``]: test/actions.html#wait -[``]: test/actions.html#waitforelement -[``]: test/actions.html#waitforelementvisible -[``]: test/actions.html#waitforloadingmasktodisappear -[Action group]: test/action-groups.html -[annotations]: test/annotations.html -[entity]: data.html -[extension]: extending.html -[merging]: merging.html -[parameterized selectors]: section/parameterized-selectors.html -[sections]: section.html diff --git a/docs/commands/codeception.md b/docs/commands/codeception.md deleted file mode 100644 index 8e6d76c11..000000000 --- a/docs/commands/codeception.md +++ /dev/null @@ -1,93 +0,0 @@ -# CLI commands: vendor/bin/codecept - -
-We do not recommend using Codeception commands directly as they can break MFTF basic workflow. -All the Codeception commands you need are wrapped using the [mftf tool][]. - -To run the Codeception testing framework commands directly, change your directory to the ``. -
- -## Usage examples - -Run all the generated tests: - -```bash -vendor/bin/codecept run functional -c dev/tests/acceptance/codeception.yml -``` - -Run all tests without the `` [annotation][]: - -```bash -vendor/bin/codecept run functional --skip-group skip -c dev/tests/acceptance/codeception.yml -``` - -Run all tests with the `` [annotation][] but with no ``: - -```bash -vendor/bin/codecept run functional --group example --skip-group skip -c dev/tests/acceptance/codeception.yml -``` - -## `codecept run` - -`codecept run` runs the test suites: - -```bash -vendor/bin/codecept run -``` - -
-The following documentation corresponds to Codeception 4.1.4. -
- -```bash -Full reference: - -Arguments: - suite suite to be tested - test test to be run - -Options: - -o, --override=OVERRIDE Override config values (multiple values allowed) - -e, --ext=EXT Run with extension enabled (multiple values allowed) - --report Show output in compact style - --html[=HTML] Generate html with results [default: "report.html"] - --xml[=XML] Generate JUnit XML Log [default: "report.xml"] - --phpunit-xml[=PHPUNIT-XML] Generate PhpUnit XML Log [default: "phpunit-report.xml"] - --tap[=TAP] Generate Tap Log [default: "report.tap.log"] - --json[=JSON] Generate Json Log [default: "report.json"] - --colors Use colors in output - --no-colors Force no colors in output (useful to override config file) - --silent Only outputs suite names and final results - --steps Show steps in output - -d, --debug Show debug and scenario output - --bootstrap[=BOOTSTRAP] Execute custom PHP script before running tests. Path can be absolute or relative to current working directory [default: false] - --no-redirect Do not redirect to Composer-installed version in vendor/codeception - --coverage[=COVERAGE] Run with code coverage - --coverage-html[=COVERAGE-HTML] Generate CodeCoverage HTML report in path - --coverage-xml[=COVERAGE-XML] Generate CodeCoverage XML report in file - --coverage-text[=COVERAGE-TEXT] Generate CodeCoverage text report in file - --coverage-crap4j[=COVERAGE-CRAP4J] Generate CodeCoverage report in Crap4J XML format - --coverage-phpunit[=COVERAGE-PHPUNIT] Generate CodeCoverage PHPUnit report in path - --no-exit Don't finish with exit code - -g, --group=GROUP Groups of tests to be executed (multiple values allowed) - -s, --skip=SKIP Skip selected suites (multiple values allowed) - -x, --skip-group=SKIP-GROUP Skip selected groups (multiple values allowed) - --env=ENV Run tests in selected environments. (multiple values allowed) - -f, --fail-fast Stop after first failure - --no-rebuild Do not rebuild actor classes on start - --seed=SEED Define random seed for shuffle setting - --no-artifacts Don't report about artifacts - -h, --help Display this help message - -q, --quiet Do not output any message - -V, --version Display this application version - --ansi Force ANSI output - --no-ansi Disable ANSI output - -n, --no-interaction Do not ask any interactive question - -c, --config[=CONFIG] Use custom path for config - -v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug -``` - - - -[mftf tool]: mftf.md -[annotation]: ../test/annotations.md \ No newline at end of file diff --git a/docs/commands/mftf.md b/docs/commands/mftf.md deleted file mode 100644 index 4b4bb4bab..000000000 --- a/docs/commands/mftf.md +++ /dev/null @@ -1,683 +0,0 @@ -# CLI commands: vendor/bin/mftf - -The Magento Functional Testing Framework (MFTF) introduces the command line interface (CLI) tool `vendor/bin/mftf` to facilitate your interaction with the framework. - -Note that `mftf` commands replace the `robo` commands that were used in previous releases. - -## Command format - -In the project root directory (where you have installed the framework as a composer dependency), run commands using the following format: - -```bash -vendor/bin/mftf command [options] [] [--remove|-r] -``` - -## Useful commands - -Use the following commands to run commonly performed tasks. - -### Apply the configuration parameters - -```bash -vendor/bin/mftf build:project -``` - -### Upgrade the project - -```bash -vendor/bin/mftf build:project --upgrade -``` - -Upgrades all installed MFTF tests after a major MFTF upgrade. - -### Generate all tests - -```bash -vendor/bin/mftf generate:tests -``` - -### Generate tests by test name - -```bash -vendor/bin/mftf generate:tests AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest -``` - -### Generate tests by testNames.txt file - -```bash -vendor/bin/mftf generate:tests -p path/to/your/testNames.txt -``` - -This command generate all tests specified in a testNames.txt file. - -#### Example - -```bash -testName1 -testName2 -testNameN -suiteName:testInSuite -``` - -### Generate test by test and suite name - -```bash -vendor/bin/mftf generate:tests WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest -``` - -### Generate test dependencies - -```bash -vendor/bin/mftf generate:tests -l testEntityJson -``` - -This command generate json file consist of all test dependent module. - -### Generate test dependencies by test name - -```bash -vendor/bin/mftf generate:tests testName1 testName2 .. testNameN -l testEntityJson -``` - -### Generate and run the tests for a specified group - -```bash -vendor/bin/mftf run:group product -r -``` - -This command cleans up the previously generated tests; generates and runs tests for the product group (where `group="product"`). - -### Generate and run particular tests - -```bash -vendor/bin/mftf run:test AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest -r -``` - -This command cleans up the previously generated tests; generates and runs the `AdminLoginSuccessfulTest` and `StorefrontPersistedCustomerLoginTest` tests. - -### Generate and run particular test in a specific suite's context - -```bash -vendor/bin/mftf run:test WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest -r -``` - -This command cleans up previously generated tests; generates and run `AdminCMSPageCreatePageTest` within the context of the `WYSIWYGDisabledSuite`. - -### Generate and run a testManifest.txt file - -```bash -vendor/bin/mftf run:manifest path/to/your/testManifest.txt -``` - -This command runs all tests specified in a `testManifest.txt` file. When you generate tests, a `testManifest.txt` file is also generated for you. You can pass this file directly to the `run:manifest` command and it will execute all listed tests. You can also create your own file in the same format to execute a subset of tests. Note: This command does not generate tests. - -### Generate previously failed tests - -```bash -vendor/bin/mftf generate:failed -``` - -This command cleans up the previously generated tests; generates the tests listed in `dev/tests/acceptance/tests/_output/failed`. -For more details about `failed`, refer to [Reporting][]. - -### Run previously failed tests - -```bash -vendor/bin/mftf run:failed -``` - -This command runs the tests listed in `dev/tests/acceptance/tests/_output/failed`. -For more details about `failed`, refer to [Reporting][]. - -## Error tolerance during generation - -Starting from version 3.2.0, MFTF will not fail right away when encountering generation errors. -Instead, MFTF will generate as many tests and suites as it can, log errors to `mftf.log`, and exit with a non-zero generation status. - -Note: -- Not all errors are tolerable at generation. For example, schema validation errors, parser errors, and WebApi authentication errors will cause `hard` failures, with no tests or suites being generated. -- Error tolerance in generation is meant to help local test development and testing and is expected to be run locally. All generation errors must be fixed in order to use other framework functionality, pass static checks, and to deliver MFTF tests. - -## Reference - -### `build:project` - -#### Description - -Clone the example configuration files and build the Codeception project. - -#### Usage - -```bash -vendor/bin/mftf build:project [--upgrade] [config_param_options] -``` - -#### Options - -| Option | Description | -| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `-u`, `--upgrade` | Upgrades all installed MFTF tests according to requirements of the last major release. Specifying this flag upgrades only those tests in the default location. Example: `build:project --upgrade`. | - -You can include options to set configuration parameter values for your environment since the project build process also [sets up the environment][setup]. - -```bash -vendor/bin/mftf build:project --MAGENTO_BASE_URL=http://magento.local/ --MAGENTO_BACKEND_NAME=admin214365 -``` - -### `doctor` - -#### Description - -Diagnose MFTF configuration and setup. Currently this command will check the following: -- Verify admin credentials are valid. Allowing MFTF authenticates and runs API requests to Magento through cURL -- Verify that Selenium is up and running and available for MFTF -- Verify that new session of browser can open Magento admin and store front urls -- Verify that MFTF can run MagentoCLI commands - -#### Usage - -```bash -vendor/bin/mftf doctor -``` - -#### Options - -### `generate:tests` - -#### Description - -Perform XML schema validation and generate PHP code from the tests defined in XML files. -The path is set in the `TESTS_MODULE_PATH` [configuration] parameter. - -#### Usage - -```bash -vendor/bin/mftf generate:tests [option] [] [] [--remove] -``` - -#### Options - -| Option | Description | -|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `--config=[ or or ]` | Creates a single manifest file with a list of all tests. The default location is `tests/functional/Magento/FunctionalTest/_generated/testManifest.txt`.
You can split the list into multiple groups using `--config=parallel`; the groups will be generated in `_generated/groups/` like `_generated/groups/group1.txt, group2.txt, ...`.
Available values: `default` (default), `singleRun`(same as `default`), and `parallel`.
Example: `generate:tests --config=parallel`. | -| `--filter` | Option to filter tests to be generated.
Template: '<filterName>:<filterValue>'.
Existing filter types: severity, includeGroup, excludeGroup.
Existing severity values: BLOCKER, CRITICAL, MAJOR, AVERAGE, MINOR.
Example: `vendor/bin/mftf generate:tests --filter=severity:CRITICAL --filter=severity:BLOCKER --filter=includeGroup:customer` | -| `--force` | Forces test generation, regardless of the module merge order defined in the Magento instance. Example: `generate:tests --force`. | -| `-i,--time` | Set time in minutes to determine the group size when `--config=parallel` is used.
Example: `generate:tests --config=parallel --time=15`
Option `--time` will be the default and the __default value__ is `10` when neither `--time` nor `--groups` is specified.
Example: `generate:tests --config=parallel` | -| `-g,--groups` | Set number of groups to be split into when `--config=parallel` is used.
Example: `generate:tests --config=parallel --groups=300`
Options `--time` and `--groups` are mutually exclusive and only one should be used. | -| `--tests` | Defines the test configuration as a JSON string or JSON file path. | -| `--allow-skipped` | Allows MFTF to generate and run tests marked with `.` | -| `--debug` | Performs schema validations on XML files.
DEFAULT: `generate:tests` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered.
DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test generation fails because of an invalid XML schema. This option takes extra processing time. Use it after test generation has failed once.
| -| `-r,--remove` | Removes the existing generated suites and tests cleaning up the `_generated` directory before the actual run. For example, `generate:tests SampleTest --remove` cleans up the entire `_generated` directory and generates `SampleTest` only. | -| `-l,--log` | Generate metadata files during test generation. Accepted parameters are: testEntityJson. | - -#### Examples of the JSON configuration - -The configuration to generate a single test with no suites: - -```json -{ - "tests":[ - "general_test1" //Generate the "general_test1" test. - ], - "suites": null -} -``` - -The configuration to generate a single test in the suite: - -```json -{ - "tests": null, // No tests outside the suite configuration will be generated. - "suites":{ - "sample":[ // The suite that contains the test. - "suite_test1" // The test to be generated. - ] - } -} -``` - -Complex configuration to generate a few non-suite tests, a single test in a suite, and an entire suite: - -```json -{ - "tests":[ - "general_test1", - "general_test2", - "general_test3" - ], - "suites":{ //Go to suites. - "sample":[ //Go to the "sample" suite. - "suite_test1" //Generate the "suite_test1" test. - ], - "sample2":[] //Generate all tests in the "sample2" suite. - } -} -``` - -The command that encodes this complex configuration: - -Command to generate test by json string: - -```bash -vendor/bin/mftf generate:tests --tests '{"tests":["general_test1","general_test2","general_test3"],"suites":{"sample":["suite_test1"],"sample2":null}}' -``` - -Note that the strings must be escaped and surrounded in quotes. - -Command to generate test by json file: - -```bash -vendor/bin/mftf generate:tests --tests ./foldername/filename.json -``` - -### `generate:suite` - -#### Description - -Generates one or more suites based on XML declarations. - -#### Usage - -```bash -vendor/bin/mftf generate:suite [] [--remove] -``` - -#### Options - -| Option | Description | -| --- | --- | -| `-r,--remove` | Removes the existing generated suites and tests cleaning up the `_generated` directory before the actual run. For example, `vendor/bin/mftf generate:suite WYSIWYG --remove` cleans up the entire `_generated` directory and generates `WYSIWYG` only. | - -#### Example - -```bash -vendor/bin/mftf generate:suite suite1 suite2 -``` - -### `generate:urn-catalog` - -#### Description - -Generates a URN catalog, enabling PhpStorm to recognize and highlight URNs. -It also enables auto-completion in PhpStorm. - -#### Usage - -```bash -vendor/bin/mftf generate:urn-catalog [--force] [] -``` - -`misc.xml` is typically located at `/.idea/misc.xml`. - -#### Options - -| Option | Description | -| ------------- | --------------------------------------------------------------------- | -| `-f, --force` | Creates the `misc.xml` file if it does not exist in the given `path`. | - -#### Example - -```bash -vendor/bin/mftf generate:urn-catalog .idea/misc.xml -``` - -### `reset` - -#### Description - -Cleans any configuration files and generated artifacts from the environment. -The `.env` file is not affected. - -#### Usage - -```bash -vendor/bin/mftf reset [--hard] -``` - -#### Options - -| Option | Description | -| -------- | ------------------------------------------ | -| `--hard` | Forces a reset of the configuration files. | - -#### Example - -```bash -vendor/bin/mftf reset --hard -``` - -### `run:group` - -Generates and executes the listed groups of tests using Codeception. - -#### Usage - -```bash -vendor/bin/mftf run:group [--skip-generate|--remove|--xml] [--] [] -``` - -#### Options - -| Option | Description | -| --------------------- | --------------------------------------------------------------------------------------------------------- | -| `-k, --skip-generate` | Skips generating from the source XML. Instead, the command executes previously-generated groups of tests. | -| `-r, --remove` | Removes previously generated suites and tests before the actual generation and run. | -| `--debug` | Performs schema validations on XML files. `run:group` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred).| -| `--xml` | Generate JUnit XML Log (default: "report.xml") | - -#### Examples - -Clean up after the last test run; generate from XML and execute the tests with the annotations `group="group1"` and `group="group2"`: - -```bash -vendor/bin/mftf -r -- run:group group1 group2 -``` - -Execute previously generated tests with the annotations `group="group1"` and `group="group2"`, skipping the regeneration of the test: - -```bash -vendor/bin/mftf run:group -k -- group1 group2 -``` - -### `run:test` - -Generates and executes tests by name using Codeception. - -#### Usage - -```bash -vendor/bin/mftf run:test [--skip-generate|--remove|--xml] [--] [] -``` - -#### Options - -| Option | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------------| -| `-k, --skip-generate` | Skips generating from the source XML. Instead, the command executes previously-generated groups of tests. | -| `-r, --remove` | Remove previously generated suites and tests. | -| `--debug` | Performs schema validations on XML files. `run:test` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred).| -| `--xml` | Generate JUnit XML Log (default: "report.xml") | - -#### Examples - -Generate the `LoginCustomerTest` and `StorefrontCreateCustomerTest` tests from XML and execute all the generated tests: - -```bash -vendor/bin/mftf run:test LoginCustomerTest StorefrontCreateCustomerTest -``` - -### `run:manifest` - -Runs a testManifest.txt file. - -This command runs all tests specified in a testManifest.xml file. It does not generate tests for you. You must do that as first. - -#### Usage - -```bash -vendor/bin/mftf run:manifest path/to/your/testManifest.txt -``` - -#### Example testManifest.xml file - -Each line should contain either: one test path or one group (-g) reference. - -``` -tests/functional/tests/MFTF/_generated/default/AdminLoginSuccessfulTestCest.php --g PaypalTestSuite -tests/functional/tests/MFTF/_generated/default/SomeOtherTestCest.php -tests/functional/tests/MFTF/_generated/default/ThirdTestCest.php --g SomeOtherSuite -``` - -### `run:failed` - -Regenerates and reruns tests that previously failed. - -This command cleans up previously generated tests. It generates and runs the tests listed in `dev/tests/acceptance/tests/_output/failed`. -For more details about `failed`, refer to [Reporting][]. - -#### Usage - -```bash -vendor/bin/mftf run:failed -``` -#### Options - -| Option | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------------| -| `--debug` | Performs schema validations on XML files. `run:failed` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred). Use it after test run has failed once.| - -#### Examples - -Run the tests that failed in the previous run: - -```bash -vendor/bin/mftf run:failed -``` - -### `setup:env` - -Updates the [configuration] parameter values in the [`.env`] file. -Creates the `.env` file if it does not exist. - -#### Usage - -```bash -vendor/bin/mftf setup:env [config_param_option1=] [config_param_option2=] -``` - -`config_param` is a configuration parameter from the `.env` file. -The command consumes the parameters in a format of options assigned with values, for example `--MAGENTO_BASE_URL=http://magento.local/`. -If you specify a parameter that the `.env` file does not contain, the command returns an error. - -You can also update configuration parameter values when you use the [`build:project`][build] command. - -#### Examples - -To change values for the `MAGENTO_BASE_URL` and `BROWSER`: - -```bash -vendor/bin/mftf setup:env --MAGENTO_BASE_URL=http://magento.local/ --BROWSER=firefox -``` - -To create a `.env` file with example parameters: - -```bash -vendor/bin/mftf setup:env -``` - -The example parameters are taken from the `etc/config/.env.example` file. - -### `static-checks` - -Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to. -Behavior for determining what tests to run is as follows: - -* If test names are specified, only those tests are run. -* If no test names are specified, tests are run according to `staticRuleset.json`. -* If no `staticRuleset.json` is found, all tests are run. - -Static checks errors are written to *.txt files under TEST_BP/tests/_output/static-results/ - -#### Usage - -```bash -vendor/bin/mftf static-checks []... -``` - -#### Options - -| Option | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------------| -| `-p, --path` | Path to a MFTF test module to run "deprecatedEntityUsage" and "pauseActionUsage" static check scripts. Option is ignored by other static check scripts. - -#### Examples - -To check what existing static check scripts are available - -```bash -vendor/bin/mftf static-checks --help -``` - -To run all existing static check scripts - -```bash -vendor/bin/mftf static-checks -``` - -To run specific static check scripts - -```bash -vendor/bin/mftf static-checks testDependencies -``` - -```bash -vendor/bin/mftf static-checks actionGroupArguments -``` - -```bash -vendor/bin/mftf static-checks deprecatedEntityUsage -``` - -```bash -vendor/bin/mftf static-checks pauseActionUsage -``` - -```bash -vendor/bin/mftf static-checks annotations -``` - -```bash -vendor/bin/mftf static-checks deprecatedEntityUsage -p path/to/mftf/test/module -``` - -```bash -vendor/bin/mftf static-checks pauseActionUsage -p path/to/mftf/test/module -``` - -```bash -vendor/bin/mftf static-checks testDependencies actionGroupArguments -``` - -#### Existing static checks - -| Argument | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------------| -|`testDependencies` | Checks that test dependencies do not violate Magento module's composer dependencies.| -|`actionGroupArguments` | Checks that action groups do not have unused arguments.| -|`deprecatedEntityUsage`| Checks that deprecated test entities are not being referenced.| -|`annotations`| Checks various details of test annotations, such as missing annotations or duplicate annotations.| -|`pauseUsage`| Checks that pause action is not used in action groups, tests or suites.| - -#### Defining ruleset - -The `static-checks` command will look for a `staticRuleset.json` file under either: - -* `dev/tests/acceptance/staticRuleset.json`, if embedded with Magento2 -* `dev/staticRuleset.json`, if standalone - -This file works as the default configuration to easily allow for the integration of `static-checks` in a CI environment. -Currently, the ruleset only defines the tests to run. Here is an example of the expected format: - -```json -{ - "tests": [ - "actionGroupArguments", - "anotherTest" - ] -} -``` - -### `upgrade:tests` - -When the path argument is specified, this `upgrade` command applies all the major version MFTF upgrade scripts to a `Test Module` in the given path. -Otherwise, it will apply all the major version MFTF upgrade scripts to all installed test components. - -`Test Module` should have the directory structure of ActionGroup, Data, Metadata, Page, Section, Test, and Suite. - -**Note**: - -The upgrade scripts are meant to be used for Test Modules under source code control. If you have old versions of test modules under vendor, those test modules will get upgraded - -#### Usage - -```bash -vendor/bin/mftf upgrade:tests [] -``` - -`` is the path to a MFTF `Test Module` that needs to be upgraded. -The command searches recursively for any `*.xml` files to upgrade. - -#### Examples - -To upgrade all installed MFTF tests: - -```bash -vendor/bin/mftf upgrade:tests -``` - -To upgrade all test components inside modules in the `dev/tests/acceptance/tests/` directory: - -```bash -vendor/bin/mftf upgrade:tests /Users/user/magento2/dev/tests/acceptance/tests/ -``` - -To upgrade all test components inside the `Catalog` module: - -```bash -vendor/bin/mftf upgrade:tests /Users/user/magento2/app/code/Magento/Catalog/Test/Mftf/ -``` - -### `codecept:run` - -A MFTF wrapper command that invokes `vendor/bin/codecept run`. This command runs tests in functional suite. Tests must be generated before using this command. - -#### Usage - -See the [Run Command](https://codeception.com/docs/reference/Commands#Run). - -```bash -vendor/bin/mftf codecept:run [] --[] -``` - -#### Examples - -```bash -# Run all tests in functional suite -vendor/bin/mftf codecept:run functional -``` - -```bash -# Run all tests in functional suite with options -vendor/bin/mftf codecept:run functional --verbose --steps --debug -``` - -```bash -# Run one test -vendor/bin/mftf codecept:run functional Magento/_generated/default/AdminCreateCmsPageTestCest.php --debug -``` - -```bash -# Run all tests in default group -vendor/bin/mftf codecept:run functional --verbose --steps -g default -``` - -
-

-Note: You may want to limit the usage of this Codeception command with arguments and options for "acceptance" only, since it is what's supported by MFTF. -When using this command, you should change "acceptance" to "functional" when referring to Codeception documentation. -

-
- - - -[configuration]: ../configuration.md -[Reference]: #reference -[build]: #buildproject -[setup]: #setupenv -[Reporting]: ../reporting.md - - - -*[MFTF]: Magento Functional Testing Framework diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index 501b1d5e9..000000000 --- a/docs/configuration.md +++ /dev/null @@ -1,446 +0,0 @@ -# Configuration - -The `*.env` file provides additional configuration for the Magento Functional Testing Framework (MFTF). -To run MFTF on your Magento instance, specify the basic configuration values. -Advanced users can create custom configurations based on requirements and environment. - -## Basic configuration - -These basic configuration values are __required__ and must be set by the user before MFTF can function correctly. - -### MAGENTO_BASE_URL - -The root URL of the Magento application under test. - -Example: - -```conf -MAGENTO_BASE_URL=http://magento2.vagrant251 -``` - -
-If the `MAGENTO_BASE_URL` contains a subdirectory (like `http://magento.test/magento2ce`), specify [`MAGENTO_CLI_COMMAND_PATH`][]. -
- -### MAGENTO_BACKEND_NAME - -The path to the Magento Admin page. - -Example: - -```conf -MAGENTO_BACKEND_NAME=admin_12346 -``` - -### MAGENTO_BACKEND_BASE_URL - -(Optional) If you are running the Admin Panel on a separate domain, specify this value: - -Example: - -```conf -MAGENTO_BACKEND_BASE_URL=https://admin.magento2.test -``` - -### MAGENTO_ADMIN_USERNAME - -The username that tests can use to access the Magento Admin page - -Example: - -```conf -MAGENTO_ADMIN_USERNAME=admin -``` - -### MAGENTO_ADMIN_PASSWORD - -The password that tests will use to log in to the Magento Admin page. - -Example: - -```conf -MAGENTO_ADMIN_PASSWORD=1234reTyt%$7 -``` - -## Advanced configuration - -Depending on the environment you use, you may need to configure MFTF more precisely by setting additional configuration parameters. -This section describes available configuration parameters and their default values (where applicable). - -### DEFAULT_TIMEZONE - -Sets a default value for the `timezone` attribute of a [`generateDate` action][generateDate]. -This value is applied when a test step does not specify a time zone. -For the complete list of available time zones, refer to [List of Supported Timezones][timezones]. - -Default: `America/Los_Angeles`. - -Example: - -```conf -DEFAULT_TIMEZONE=UTC -``` - -### SELENIUM - -The `SELENIUM_*` values form the URL of a custom Selenium server for running testing. - -Default Selenium URL: `http://127.0.0.1:4444/wd/hub` - -And the default configuration: - -```conf -SELENIUM_HOST=127.0.0.1 -SELENIUM_PORT=4444 -SELENIUM_PROTOCOL=http -SELENIUM_PATH=/wd/hub -``` - -
-`SELENIUM_*` values are required if you are running Selenium on an external system. -If you change the configuration of the external Selenium server, you must update these values. -
- -#### SELENIUM_HOST - -Override the default Selenium server host. - -Example: - -```conf -SELENIUM_HOST=user:pass@ondemand.saucelabs.com -``` - -#### SELENIUM_PORT - -Override the default Selenium server port. - -Example: - -```conf -SELENIUM_PORT=443 -``` - -#### SELENIUM_PROTOCOL - -Override the default Selenium server protocol. - -Example: - -```conf -SELENIUM_PROTOCOL=https -``` - -#### SELENIUM_PATH - -Override the default Selenium server path. - -Example: - -```conf -SELENIUM_PATH=/wd/hub -``` - -### MAGENTO_RESTAPI - -These `MAGENTO_RESTAPI_*` values are optional and can be used in cases when your Magento instance has a different API path than the one in `MAGENTO_BASE_URL`. - -```conf -MAGENTO_RESTAPI_SERVER_HOST -MAGENTO_RESTAPI_SERVER_PORT -``` - -#### MAGENTO_RESTAPI_SERVER_HOST - -The protocol and the host of the REST API server path. - -Example: - -```conf -MAGENTO_RESTAPI_SERVER_HOST=http://localhost -``` - -#### MAGENTO_RESTAPI_SERVER_PORT - -The port part of the API path. - -Example: - -```conf -MAGENTO_RESTAPI_SERVER_PORT=5000 -``` - -### \*_BP - -Settings to override base paths for the framework. -You can use it when MFTF is applied as a separate tool. -For example, when you need to place MFTF and the Magento codebase in separate projects. - -```conf -MAGENTO_BP -TESTS_BP -FW_BP -TESTS_MODULES_PATH -``` - -#### MAGENTO_BP - -The path to a local Magento codebase. -It enables the [`bin/mftf`][mftf] commands such as `run` and `generate` to parse all modules of the Magento codebase for MFTF tests. - -```conf -MAGENTO_BP=~/magento2/ -``` - -#### TESTS_BP - -BP is an acronym for _Base Path_. -The path to where MFTF supplementary files are located in the Magento codebase. - -Example: - -```conf -TESTS_BP=~/magento2ce/dev/tests/acceptance -``` - -#### FW_BP - -The path to MFTF. -FW_BP is an acronym for _FrameWork Base Path_. - -Example: - -```conf -FW_BP=~/magento/magento2-functional-testing-framework -``` - -### TESTS_MODULE_PATH - -The path to where the MFTF modules mirror Magento modules. - -Example: - -```conf -TESTS_MODULE_PATH=~/magento2/dev/tests/acceptance/tests/functional/Magento -``` - -### MODULE_ALLOWLIST - -Use for a new module. -When adding a new directory at `tests/functional/Magento`, add the directory name to `MODULE_ALLOWLIST` to enable MFTF to process it. - -Example: - -```conf -MODULE_ALLOWLIST=Magento_Framework,Magento_ConfigurableProductWishlist,Magento_ConfigurableProductCatalogSearch -``` - -### MAGENTO_CLI_COMMAND_PATH - -Path to the Magento CLI command entry point. - -Default: `dev/tests/acceptance/utils/command.php`. -It points to `MAGENTO_BASE_URL` + `dev/tests/acceptance/utils/command.php` - -Modify the default value: - -- for non-default Magento installation -- when using a subdirectory in the `MAGENTO_BASE_URL` - -Example: `dev/tests/acceptance/utils/command.php` - -### BROWSER - -Override the default browser performing the tests. - -Default: Chrome - -Example: - -```conf -BROWSER=firefox -``` - -### CREDENTIAL_VAULT_ADDRESS - -The Api address for a vault server. - -Default: http://127.0.0.1:8200 - -Example: - -```conf -# Default api address for local vault dev server -CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 -``` - -### CREDENTIAL_VAULT_SECRET_BASE_PATH - -Vault secret engine base path. - -Default: secret - -Example: - -```conf -# Default base path for kv secret engine in local vault dev server -CREDENTIAL_VAULT_SECRET_BASE_PATH=secret -``` - -### CREDENTIAL_AWS_SECRETS_MANAGER_REGION - -The region that AWS Secrets Manager is located. - -Example: - -```conf -# Region of AWS Secrets Manager -CREDENTIAL_AWS_SECRETS_MANAGER_REGION=us-east-1 -``` - -### CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE - -The profile used to connect to AWS Secrets Manager. - -Example: - -```conf -# Profile used to connect to AWS Secrets Manager. -CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE=default -``` - -### VERBOSE_ARTIFACTS - -Determines if passed tests should still have all their Allure artifacts. These artifacts include `.txt` attachments for `dontSee` actions and `createData` actions. - -If enabled, all tests will have all of their normal Allure artifacts. - -If disabled, passed tests will have their Allure artifacts trimmed. Failed tests will still contain all their artifacts. - -This is set `false` by default. - -```conf -VERBOSE_ARTIFACTS=true -``` - -### ENABLE_BROWSER_LOG - -Enables addition of browser logs to Allure steps - -```conf -ENABLE_BROWSER_LOG=true -``` - -### SELENIUM_CLOSE_ALL_SESSIONS - -Forces MFTF to close all Selenium sessions after running a suite. - -Use this if you're having issues with sessions hanging in an MFTF suite. - -```conf -SELENIUM_CLOSE_ALL_SESSIONS=true -``` - -### BROWSER_LOG_BLOCKLIST - -Blocklists types of browser log entries from appearing in Allure steps. - -Denoted in browser log entry as `"SOURCE": "type"`. - -```conf -BROWSER_LOG_BLOCKLIST=other,console-api -``` - -### WAIT_TIMEOUT - -Global MFTF configuration for the default amount of time (in seconds) that a test will wait while loading a page. - -```conf -WAIT_TIMEOUT=30 -``` - -### ENABLE_PAUSE - -Enables the ability to pause test execution at any point, and enter an interactive shell where you can try commands in action. -When pause is enabled, MFTF will generate pause() command in _failed() hook so that test will pause execution when failed. - -```conf -ENABLE_PAUSE=true -``` - -### REMOTE_STORAGE_AWSS3_DRIVER - -The remote storage driver. To enable AWS S3, use `aws-s3`. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_DRIVER=aws-s3 -``` - -### REMOTE_STORAGE_AWSS3_REGION - -The region of S3 bucket. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_REGION=us-west-2 -``` - -### REMOTE_STORAGE_AWSS3_BUCKET - -The name of S3 bucket. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_BUCKET=my-test-bucket -``` - -### REMOTE_STORAGE_AWSS3_PREFIX - -The optional prefix inside S3 bucket. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_PREFIX=local -``` - -### REMOTE_STORAGE_AWSS3_ACCESS_KEY - -The optional access key for the S3 bucket. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_ACCESS_KEY=access-key -``` - -### REMOTE_STORAGE_AWSS3_SECRET_KEY - -The optional secret key for the S3 bucket. - -Example: - -```conf -REMOTE_STORAGE_AWSS3_SECRET_KEY=secret-key -``` - -### MAGENTO_ADMIN_WEBAPI_TOKEN_LIFETIME - -The lifetime (in seconds) of Magento Admin WebAPI token; if token is older than this value a refresh attempt will be made just before the next WebAPI call. - -Example: - -```conf -MAGENTO_ADMIN_WEBAPI_TOKEN_LIFETIME=10800 -``` - - - -[`MAGENTO_CLI_COMMAND_PATH`]: #magento_cli_command_path -[generateDate]: test/actions.md#generatedate -[mftf]: commands/mftf.md -[timezones]: https://php.net/manual/en/timezones.php diff --git a/docs/configure-2fa.md b/docs/configure-2fa.md deleted file mode 100644 index 7b01913b9..000000000 --- a/docs/configure-2fa.md +++ /dev/null @@ -1,53 +0,0 @@ -# Configuring MFTF for Two-Factor Authentication (2FA) - -Using two-factor authentication (2FA) with MFTF is possible with some configurations settings in Magento. -In this document, we will use Google as the authentication provider. - -## Configure Magento {#config-magento-2fa} - -To prepare Magento for MFTF testing when 2FA is enabled, set the following configurations through the Magento CLI. - -First, select `Google Authenticator` as Magento's 2FA provider: - -```bash -bin/magento config:set twofactorauth/general/force_providers google -``` - -Now set the OTP window to `60` seconds: - -```bash -bin/magento config:set twofactorauth/google/otp_window 60 -``` - -Set a base32-encoded `secret` for `Google Authenticator` to generate a OTP for the default admin user that you set for `MAGENTO_ADMIN_USERNAME` in `.env`: - -```bash -bin/magento security:tfa:google:set-secret -``` - -## Configure the MFTF {#config-mftf-2fa} - -Save the same base32-encoded `secret` in a MFTF credential storage, e.g. `.credentials` file, `HashiCorp Vault` or `AWS Secrets Manager`. -More details are [here](./credentials.md). - -The path of the `secret` should be: - -```conf -magento/tfa/OTP_SHARED_SECRET -``` - -## GetOTP {#getOTP} - -A one-time password (OTP) is required when an admin user logs into the Magento admin. -Use the action `getOTP` [Reference](./test/actions.md#getotp) to generate the code and use it for the `Authenticator code` text field in 2FA - Google Auth page. - -Note: -You will need to set the `secret` for any non-default admin users first, before using `getOTP`. For example: - -{%raw%} - -```xml - -``` - -{%endraw%} diff --git a/docs/credentials.md b/docs/credentials.md deleted file mode 100644 index 56f1ddbc4..000000000 --- a/docs/credentials.md +++ /dev/null @@ -1,290 +0,0 @@ -# Credentials - -When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, -use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys. - -Currently MFTF supports three types of credential storage: - -- **.credentials file** -- **HashiCorp Vault** -- **AWS Secrets Manager** - -## Configure File Storage - -MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. -The file contains an example list of keys for fields that can require credentials. - -### Create `.credentials` - -To make MFTF process the file with credentials, in the command line, navigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. - -```bash -cd dev/tests/acceptance/ -``` - -```bash -cp .credentials.example .credentials -``` - -### Add `.credentials` to `.gitignore` - -Verify that the file is excluded from tracking by `.gitignore` (unless you need this behavior): - -```bash -git check-ignore .credentials -``` - -The command outputs the path if the file is excluded: - -```terminal -.credentials -``` - -### Define sensitive data in the `.credentials` file - -Open the `.credentials` file and, for Magento core credentials, uncomment the fields you want to use and add your values: - -```conf -... -# Credentials for the USPS service -magento/carriers_usps_userid=usps_test_user -magento/carriers_usps_password=Lmgxvrq89uPwECeV - -# Credentials for the DHL service -#magento/carriers_dhl_id_us=dhl_test_user -#magento/carriers_dhl_password_us=Mlgxv3dsagVeG -.... -``` -### Add key and value pair for admin password . -magento/MAGENTO_ADMIN_PASSWORD must contain the user password required for authorization in the Admin area. Example: magento/MAGENTO_ADMIN_PASSWORD=mycustompassword - -```conf -... -# Admin password -magento/MAGENTO_ADMIN_PASSWORD =123123q - -.... -``` - -Or add new key/value pairs for your own credentials. The keys use the following format: - -```conf -/= -``` - -
-The `/` symbol is not supported in a `key_name` other than the one after your vendor or extension name. -
- -Otherwise you are free to use any other `key_name` you like, as they are merely the keys to reference from your tests. - -```conf -# Credentials for the MyAwesome service -vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A -``` - -## Configure Vault Storage - -Hashicorp vault secures, stores, and tightly controls access to data in modern computing. -It provides advanced data protection for your testing credentials. - -MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. - -### Install vault CLI - -Download and install vault CLI tool if you want to run or develop MFTF tests locally. [Download Vault][Download Vault] - -### Authenticate to vault via vault CLI - -Authenticate to vault server via the vault CLI tool: [Login Vault][Login Vault]. - -```bash -vault login -method -path -``` - -**Do not** use `-no-store` command option, as MFTF will rely on the persisted token in the token helper (usually the local filesystem) for future API requests. - -### Store secrets in vault - -MFTF uses the `KV Version 2` secret engine for secret storage. -More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. - -#### Secrets path and key convention - -The path and key for secret data must follow the format: - -```conf -/mftf// -``` - -```conf -# Secret path and key for carriers_usps_userid -secret/mftf/magento/carriers_usps_userid - -# Secret path and key for carriers_usps_password -secret/mftf/magento/carriers_usps_password -``` - -#### Write secrets to vault - -You can use vault CLI or API to write secret data (credentials, etc) to vault. Here is a CLI example: - -```bash -vault kv put secret/mftf/magento/carriers_usps_userid carriers_usps_userid=usps_test_user -vault kv put secret/mftf/magento/carriers_usps_password carriers_usps_password=Lmgxvrq89uPwECeV -``` - -### Setup MFTF to use vault - -Add vault configuration environment variables [`CREDENTIAL_VAULT_ADDRESS`][] and [`CREDENTIAL_VAULT_SECRET_BASE_PATH`][] -from `etc/config/.env.example` in `.env`. -Set values according to your vault server configuration. - -```conf -# Default vault dev server -CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 -CREDENTIAL_VAULT_SECRET_BASE_PATH=secret -``` - -## Configure AWS Secrets Manager - -AWS Secrets Manager offers secret management that supports: -- Secret rotation with built-in integration for Amazon RDS, Amazon Redshift, and Amazon DocumentDB -- Fine-grained policies and permissions -- Audit secret rotation centrally for resources in the AWS Cloud, third-party services, and on-premises - -### Prerequisites - -#### Use AWS Secrets Manager from your own AWS account - -- An AWS account with Secrets Manager service -- An IAM user with AWS Secrets Manager access permission - -#### Use AWS Secrets Manager in CI/CD - -- AWS account ID where the AWS Secrets Manager service is hosted -- Authorized CI/CD EC2 instances with AWS Secrets Manager service access IAM role attached - -### Store secrets in AWS Secrets Manager - -#### Secrets format - -`Secret Name` and `Secret Value` are two key pieces of information for creating a secret. - -`Secret Value` can be either plaintext or key/value pairs in JSON format. - -`Secret Name` must use the following format: - -```conf -mftf// -``` - -`Secret Value` can be stored in two different formats: plaintext or key/value pairs. - -For plaintext format, `Secret Value` can be any string you want to secure. - -For key/value pairs format, `Secret Value` is a key/value pair with `key` the same as `Secret Name` without `mftf//` prefix, which is ``, and value can be any string you want to secure. - -##### Create Secrets using AWS CLI - -```bash -aws secretsmanager create-secret --name "mftf/magento/shipping/carriers_usps_userid" --description "Carriers USPS user id" --secret-string "1234567" -``` - -##### Create Secrets using AWS Console - -- Sign in to the AWS Secrets Manager console -- Choose Store a new secret -- In the Select secret type section, specify "Other type of secret" -- For `Secret Name`, `Secret Key` and `Secret Value` field, for example, to save the same secret in key/value JSON format, you should use - -```conf -# Secret Name -mftf/magento/shipping/carriers_usps_userid - -# Secret Key -shipping/carriers_usps_userid - -# Secret Value -1234567 -``` - -### Setup MFTF to use AWS Secrets Manager - -To use AWS Secrets Manager, the AWS region to connect to is required. You can set it through environment variable [`CREDENTIAL_AWS_SECRETS_MANAGER_REGION`][] in `.env`. - -MFTF uses the recommended [Default Credential Provider Chain][credential chain] to establish connection to AWS Secrets Manager service. -You can setup credentials according to [Default Credential Provider Chain][credential chain] and there is no MFTF specific setup required. -Optionally, however, you can explicitly set AWS profile through environment variable [`CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE`][] in `.env`. - -```conf -# Sample AWS Secrets Manager configuration -CREDENTIAL_AWS_SECRETS_MANAGER_REGION=us-east-1 -CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE=default -``` - -### Optionally set CREDENTIAL_AWS_ACCOUNT_ID environment variable - -In case AWS credentials cannot resolve to a valid AWS account, full AWS KMS ([Key Management Service][]) key ARN ([Amazon Resource Name][]) is required. -You will also need to set `CREDENTIAL_AWS_ACCOUNT_ID` environment variable so that MFTF can construct the full ARN. This is mostly used for CI/CD. - -```bash -export CREDENTIAL_AWS_ACCOUNT_ID= -``` - -## Configure multiple credential storage - -It is possible and sometimes useful to setup and use multiple credential storage at the same time. -In this case, the MFTF tests are able to read secret data at runtime from all storage options. MFTF will use the following precedence: - -``` -.credentials File > HashiCorp Vault > AWS Secrets Manager -``` - - -## Use credentials in a test - -Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`createData`][]. - -Define the value as a reference to the corresponding key in the credentials file or vault such as `{{_CREDS.my_data_key}}`: - -- `_CREDS` is an environment constant pointing to the `.credentials` file -- `my_data_key` is a key in the the `.credentials` file or vault that contains the value to be used in a test step - - for File Storage, ensure your key contains the vendor prefix, which is `vendor/my_data_key` - -For example, to reference secret data in the [`fillField`][] action, use the `userInput` attribute using a typical File Storage: - -```xml - -``` - - - -## Implementation details - -The generated tests do not contain credentials values. -MFTF dynamically retrieves, encrypts, and decrypts the sensitive data during test execution. -Decrypted credentials do not appear in the console, error logs, or [test reports][]. -The decrypted values are only available in the `.credentials` file or within vault. - -
-The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data. -
- - -[`fillField`]: test/actions.md#fillfield -[`magentoCLI`]: test/actions.md#magentocli -[`createData`]: test/actions.md#createdata -[data]: data.md -[initial setup]: getting-started.md -[test reports]: reporting.md -[Download Vault]: https://www.hashicorp.com/products/vault/ -[Login Vault]: https://www.vaultproject.io/docs/commands/login.html -[Vault KV2]: https://www.vaultproject.io/docs/secrets/kv/kv-v2.html -[`CREDENTIAL_VAULT_ADDRESS`]: configuration.md#credential_vault_address -[`CREDENTIAL_VAULT_SECRET_BASE_PATH`]: configuration.md#credential_vault_secret_base_path -[credential chain]: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials.html -[`CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE`]: configuration.md#credential_aws_secrets_manager_profile -[`CREDENTIAL_AWS_SECRETS_MANAGER_REGION`]: configuration.md#credential_aws_secrets_manager_region -[Key Management Service]: https://aws.amazon.com/kms/ -[Amazon Resource Name]: https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html diff --git a/docs/custom-helpers.md b/docs/custom-helpers.md deleted file mode 100644 index bf1ebc572..000000000 --- a/docs/custom-helpers.md +++ /dev/null @@ -1,129 +0,0 @@ -# Custom Helpers - -
-Due to complexity, you should only write new custom helpers as a last resort, after trying to implement your test using built-in actions. -
- -Custom Helpers allow test writers to write custom test actions to solve advanced requirements beyond what MFTF offers out of the box. - -In MFTF version 3.0.0, the following test actions were removed: - -* `` -* `` - -These actions were removed because they allowed custom PHP code to be written inline inside of XML files. This code was difficult to read. It had no proper syntax highlighting and no linting. It was difficult to maintain, troubleshoot, and modify. - -However, sometimes custom logic beyond what MFTF offers is necessary so we have provided an alternative solution: the `` action. - -## Example - -Custom helpers are implemented in PHP files that must be placed in this directory: - -```text -/Test/Mftf/Helper -``` - -This custom helper selects text on the page with this approach: - -1. Move to a very specific X,Y starting position. -1. Click and hold the mouse button down. -1. Move to another specific X,Y position. -1. Release the mouse button. - -This functionality is used to select text on the page and cannot be accomplished using built-in test actions. - -### PHP File - -```php -getModule('\Magento\FunctionalTestingFramework\Module\MagentoWebDriver'); - - $contextElement = $webDriver->webDriver->findElement(\Facebook\WebDriver\WebDriverBy::xpath($context)); - $actions = new \Facebook\WebDriver\Interactions\WebDriverActions($webDriver->webDriver); - $actions->moveToElement($contextElement, $startX, $startY) - ->clickAndHold() - ->moveToElement($contextElement, $endX, $endY) - ->release() - ->perform(); - } catch (\Exception $e) { - $this->fail($e->getMessage()); - } - } -} -``` - -### Notes about this PHP file - -The following details are important about the above file: - -1. The `namespace` must match the file location: `namespace Magento\PageBuilder\Test\Mftf\Helper;` -2. The class must `extends Helper` and have the corresponding `use` statement to match. -3. You may access the WebDriver object via `$this->getModule('\Magento\FunctionalTestingFramework\Module\MagentoWebDriver')`. -4. You may implement multiple related methods within the same class. -5. Specify the correct function argument types to match the type of values you want to pass in. In this case, we specified `string $context, int $startX, int $startY, int $endX, int $endY`. In the XML we will match these types. - -You should follow the same patterns in any custom helpers that you write yourself, but you may implement any logic or iteration that you need to solve for your use case. - -### Referencing in a test - -Once you have implemented something like the above PHP file, reference it in a test: - -```xml - - //div[contains(@class, 'inline-wysiwyg')]//h2 - {{TinyMCEPartialHeadingSelection.startX}} - {{TinyMCEPartialHeadingSelection.startY}} - {{TinyMCEPartialHeadingSelection.endX}} - {{TinyMCEPartialHeadingSelection.endY}} - -``` - -### Notes about the XML - -1. Specify an argument value for every argument that matches our PHP implementation. This allows us to pass other test data to the Custom Helper. -1. The `class` attribute matches the namespace specified in the PHP file. -1. Specify the method from the class via the `method` attribute. -1. If the function has a return value, it will be assigned to the `stepKey` variable. In this case, `$selectHeadingTextInTinyMCE` holds the return value. -1. The types of argument values must match the PHP implementation's expected types. - -## Key takeaways - -Custom helpers allow you to solve complex use cases such as conditional logic, iteration, or complex WebDriver usage. - -With access to the WebDriver object, you have a lot of flexibility available to you. See the [Codeception WebDriver](https://github.com/Codeception/module-webdriver/blob/master/src/Codeception/Module/WebDriver.php) for technical details and functionality available for use. - -A custom helper is written in a PHP file and then referenced in test XML, like other actions. - -You should only use these as a last resort after trying to implement your test using built-in actions. - -## References - -[Codeception WebDriver source code](https://github.com/Codeception/module-webdriver/blob/master/src/Codeception/Module/WebDriver.php) - Reference for using the WebDriver Object diff --git a/docs/data.md b/docs/data.md deleted file mode 100644 index 721f94a7e..000000000 --- a/docs/data.md +++ /dev/null @@ -1,345 +0,0 @@ -# Input testing data - -MFTF enables you to specify and use `` entities defined in XML. Default `` entities are provided for use and as templates for entity creation and manipulation. -The following diagram shows the XML structure of an MFTF data object: - -![MFTF Data Object](img/data-dia.svg) - - - -The MFTF `` entities are stored in `/Test/Mftf/Data/`. - -## Supply data to test by reference to a data entity - -Test steps requiring `` input in an action, like filling a field with a string, may reference an attribute from a data entity: - -```xml -userInput="{{SimpleSubCategory.name}}" -``` - -In this example: - -* `SimpleSubCategory` is an entity name. -* `name` is a `` key of the entity. The corresponding value will be assigned to `userInput` as a result. - -The following is an example of the usage of `` entity in the `Magento/Customer/Test/Mftf/Test/AdminCustomersAllCustomersNavigateMenuTest.xml` test: - -```xml - - - - -``` - -In the above example: - -* `AdminMenuCustomers` is an entity name. -* `dataUiId` is a `` key of the entity. - -### Environmental data - -```xml -userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" -``` - -In this example: - -* `_ENV` is a reference to the `dev/tests/acceptance/.env` file, where basic environment variables are set. -* `MAGENTO_ADMIN_USERNAME` is a name of an environment variable. - The corresponding value will be assigned to `userInput` as a result. - -The following is an example of the usage of `_ENV` in the `Magento/Braintree/Test/Mftf/ActionGroup/AdminDeleteRoleActionGroup.xml` action group: - -```xml - -``` - -### Sensitive data - -```xml -userInput="{{_CREDS.my_secret_token}}" -``` - -In this example: - -* `_CREDS` is a constant to reference to the `dev/tests/acceptance/.credentials` file, where sensitive data and secrets are stored for use in a test. -* `MY_SECRET_TOKEN` is the name of a key in the credentials variable. - The corresponding value of the credential will be assigned to `userInput` as a result. -* The decrypted values are only available in the `.credentials` file in which they are stored. - -Learn more in [Credentials][]. - -The following is an example of the usage of `_CREDS` in the `Magento/Braintree/Test/Mftf/Data/BraintreeData.xml` data entity: - -```xml - - {{_CREDS.magento/braintree_enabled_fraud_merchant_id}} - -``` - -## Persist a data entity as a prerequisite of a test {#persist-data} - -A test can specify an entity to be persisted (created in the database) so that the test actions could operate on the existing known data. - -Example of referencing `data` in a test: - -```xml -userInput="$createCustomer.email$" -``` - -In this example: - -* `createCustomer` is a step key of the corresponding test step that creates an entity. -* `email` is a data key of the entity. - The corresponding value will be assigned to `userInput` as a result. - -The following is an example of the usage of the persistant data in `Magento/Customer/Test/Mftf/Test/AdminCreateCustomerWithCountryUSATest.xml` test: - -```xml - - - -``` - -
-As of MFTF 2.3.6, you no longer need to differentiate between scopes (a test, a hook, or a suite) for persisted data when referencing it in tests. -
- -MFTF now stores the persisted data and attempts to retrieve it using the combination of `stepKey` and the scope of where it has been called. -The current scope is preferred, then widening to _test > hook > suite_ or _hook > test > suite_. - -This emphasizes the practice for the `stepKey` of `createData` to be descriptive and unique, as a duplicated `stepKey` in both a `` and `` prefers the `` data. - -## Use data returned by test actions - -A test can also reference data that was returned as a result of [test actions][], like the action ` - -``` - -The following is an example of the `Magento/Catalog/Test/Mftf/ActionGroup/AssertDiscountsPercentageOfProductsActionGroup.xml` test: - -```xml - - - {{amount}} - $grabProductTierPriceInput - -``` - -## Hard-coded data input - -The data to operate against can be included as literals in a test. Hard-coded data input can be useful in assertions. - -See also [Actions][]. - -```xml -userInput="We'll email you an order confirmation with details and tracking info." -``` - -## Format - -The format of the `` entity is: - -```xml - - - - - - - - - - - -``` - -## Principles - -The following conventions apply to MFTF ``: - -* A `` file may contain multiple data entities. -* Camel case is used for `` elements. The name represents the `` type. For example, a file with customer data is `CustomerData.xml`. A file for simple product would be `SimpleProductData.xml`. -* Camel case is used for the entity name. -* The file name must have the suffix `Data.xml`. - -## Example - -Example (`Magento/Catalog/Test/Mftf/Data/CategoryData.xml` file): - -```xml - - - - - simpleCategory - simplecategory - true - - - SimpleSubCategory - simplesubcategory - true - true - - -``` - -This example declares two `` entities: `_defaultCategory` and `SimpleSubCategory`. They set the data required for [category creation][]. - -All entities that have the same name will be merged during test generation. Both entities are of the `category` type. - -`_defaultCategory` sets three data fields: - -* `name` defines the category name as `simpleCategory` with a unique suffix. Example: `simpleCategory598742365`. -* `name_lwr` defines the category name in lowercase format with a unique suffix. Example: `simplecategory697543215`. -* `is_active` sets the enable category to `true`. - -`SimpleSubCategory` sets four data fields: - -* `name` that defines the category name with a unique suffix. Example: `SimpleSubCategory458712365`. -* `name_lwr` that defines the category name in lowercase format with a unique suffix. Example: `simplesubcategory753698741`. -* `is_active` sets the enable category to `true`. -* `include_in_menu` that sets the include in the menu to `true`. - -The following is an example of a call in test: - -```xml - -``` - - - -This action inputs data from the `name` of the `_defaultCategory` entity (for example, `simpleCategory598742365`) into the field with the locator defined in the selector of the `categoryNameInput` element of the `AdminCategoryBasicFieldSection`. - -You can also call data from the xml definition of a `data` tag directly: - -```xml - - admin - {{AnotherUser.current_password}} - {{_ENV.MAGENTO_ADMIN_PASSWORD}} - -``` - -## Reference - -### entities {#entities-tag} - -`` is an element that contains all `` elements. - -### entity {#entity-tag} - -`` is an element that contains `` elements. - -Attributes|Type|Use|Description ----|---|---|--- -`name`|string|optional|Name of the ``. Use camel case for entity names. -`type`|string|optional|Node containing the exact name of `` type. Used later to find specific Persistence Layer Model class. `type` in `` can be whatever the user wants; There are no constraints. It is important when persisting data, depending on the `type` given, as it will try to match a metadata definition with the operation being done. Example: A `myCustomer` entity with `type="customer"`, calling ``, will try to find a metadata entry with the following attributes: ``. -`deprecated`|string|optional|Used to warn about the future deprecation of the data entity. String will appear in Allure reports and console output at runtime. - -`` may contain one or more [``][], [``][], [``][], or [``][] elements in any sequence. - -### data {#data-tag} - -`` is an element containing a data/value pair. - -Attributes|Type|Use|Description ----|---|---|--- -`key`|string|optional|Key attribute of data/value pair. -`unique`|enum: `"prefix"`, `"suffix"`|optional|Add suite or test wide unique sequence as "prefix" or "suffix" to the data value if specified. - -Example: - -```xml -simpleCategory -``` - -### var {#var-tag} - -`` is an element that can be used to grab a key value from another entity. For example, when creating a customer with the `` action, the server responds with the auto-incremented ID of that customer. Use `` to access that ID and use it in another data entity. - -Attributes|Type|Use|Description ----|---|---|--- -`key`|string|optional|Key attribute of this entity to assign a value to. -`entityType`|string|optional|Type attribute of referenced entity. -`entityKey`|string|optional|Key attribute of the referenced entity from which to get a value. -`unique`|--|--|*This attribute hasn't been implemented yet.* - -Example: - -```xml - -``` - -### requiredEntity {#requiredentity-tag} - -`` is an element that specifies the parent/child relationship between complex types. - -Example: You have customer address info. To specify that relationship: - -```xml - - ... - AddressEntity - ... - -``` - -Attributes|Type|Use|Description ----|---|---|--- -`type`|string|optional|Type attribute of ``. - -### array {#array-tag} - -`` is an element that contains a reference to an array of values. - -Example: - -```xml - - ... - - 7700 W Parmer Ln - Bld D - - ... - -``` - -Attributes|Type|Use|Description ----|---|---|--- -`key`|string|required|Key attribute of this entity in which to assign a value. - -`` may contain [``][] elements. - -### item {#item-tag} - -`` is an individual piece of data to be passed in as part of the parent `` type. - -Attributes|Type|Use|Description ----|---|---|--- -`name`|string|optional|Key attribute of entity in which to assign a value. By default numeric key will be generated. - - - -[``]: #array-tag -[``]: #data-tag -[``]: #item-tag -[``]: #requiredentity-tag -[``]: #var-tag -[Actions]: ./test/actions.md -[category creation]: https://docs.magento.com/user-guide/catalog/category-create.html -[Credentials]: ./credentials.md -[test actions]: ./test/actions.md#actions-returning-a-variable diff --git a/docs/debugging.md b/docs/debugging.md deleted file mode 100644 index be17e952a..000000000 --- a/docs/debugging.md +++ /dev/null @@ -1,37 +0,0 @@ -# Debugging - -Debugging within the Magento Functional Testing Framework is helpful in identifying test bugs by allowing you to pause execution so that you may: - -- Examine the page. -- Check returned data and other variables being used during run-time. - -This is straightforward to do once you create a basic Debug Configuration. - -## Prerequisites - -- [Xdebug][] -- PHPUnit configured for use in [PHPStorm][] - -## Creating Debug Configuration with PHPStorm - -1. If not already installed, download the Codeception Framework plugin for PHPStorm (`PhpStorm->Preferences->Plugins`). -1. Click `Edit Configurations` on the configuration dropdown. -1. Click `+` and select `Codeception` from the available types. -1. Change `Test Scope` to `Type` and select `functional` from the `Type:` dropdown. -1. Find the `Custom Working Directory` option and set the path to your `dev/tests/acceptance/` directory. - -If you get a warning `Path to Codeception for local machine is not configured.`: - -1. Click `Fix`, then `+`, and select `Codeception Local`. -1. Click `...` and locate `/vendor/bin/codecept` in your Magento installation folder. - -The easiest method of tagging a test for debugging is the following: - -- In your Debug configuration, locate `Test Runner options:` and set `--group testDebug`. -- When you want to debug a test you are working on, simply add `` to the annotations. Be sure to remove this after done debugging. - -Your Debug Configuration should now be able to run your test and pause execution on any breakpoints you have set in the generated `.php` file under the `_generated` folder. - - -[Xdebug]: https://xdebug.org/docs/install -[PHPStorm]: https://www.jetbrains.com/phpstorm/ diff --git a/docs/extending.md b/docs/extending.md deleted file mode 100644 index 576bfdb24..000000000 --- a/docs/extending.md +++ /dev/null @@ -1,342 +0,0 @@ -# Extending - -There are cases when you need to create many tests that are very similar to each other. -For example, only one or two parameters (for example, URL) might vary between tests. -To avoid copy-pasting and to save some time the Magento Functional Testing Framework (MFTF) enables you to extend test components such as [test], [data], and [action group]. -You can create or update any component of the parent body in your new test/action group/entity. - -* A test starting with `` creates a test `SampleTest` that takes body of existing test `ParentTest` and adds to it the body of `SampleTest`. -* An action group starting with `` creates an action group based on the `ParentActionGroup`, but with the changes specified in `SampleActionGroup`. -* An entity starting with `` creates an entity `SampleEntity` that is equivalent to merging the `SampleEntity` with the `ParentEntity`. - -Specify needed variations for a parent object and produce a copy of the original that incorporates the specified changes (the "delta"). - -
-Unlike merging, the parent test (or action group) will still exist after the test generation. -
- -
-
-Note: The extended test will be skipped if the parent test is skipped. -
- -## Extending tests - -### Update a test step - - - -__Use case__: Create two similar tests with a different action group reference by overwriting a `stepKey`. - -> Test with "extends": - -```xml - - - - - - - - - - -``` - -> Test without "extends": - -```xml - - - - - - - - - - - - -``` - -### Add a test step - -__Use case__: Create two similar tests where the second test contains two additional steps specified to occur `before` or `after` other `stepKeys`. - -> Tests with "extends": - -```xml - - - - - - - - - - - -``` - -> Tests without "extends": - -```xml - - - - - - - - - - - - - - -``` - -### Update a test before hook - -__Use case__: Create two similar tests where the second test contains an additional action in the `before` hook. - -> Tests with "extends": - -```xml - - - - - - - - - - - - - - -``` - -> Tests without "extends": - -```xml - - - - - - - - - - - - - - - - - -``` - -## Extending action groups - -Extend an [action group] to add or update [actions] in your module. - -### Update an action - -__Use case__: The `AssertAdminCountProductActionGroup` action group counts the particular product. -Modify the action group to use another product. - -> Action groups with "extends": - -```xml - - - - - - - - {{count}} - grabProducts - - - - - - - -``` - -> Action groups without "extends": - -```xml - - - - - - - - {{count}} - grabProducts - - - - - - - - - - {{count}} - grabProducts - - - -``` - -### Add an action - -__Use case__: The `AdminGetProductCountActionGroup` action group returns the count of products. -Add a new test `AssertAdminVerifyProductCountActionGroup` that asserts the count of products: - -> Action groups with "extends": - -```xml - - - - - - - - - - - - - - {{count}} - grabProducts - - - -``` - -> Action groups without "extends": - -```xml - - - - - - - - - - - - - - - - {{count}} - grabProducts - - - -``` - - - -## Extending data - -Extend data to reuse entities in your module. - -### Update a data entry - -__Use case__: Create an entity named `DivPanelGreen`, which is similar to the `DivPanel` entity, except that it is green. - -> Entities with "extends": - -```xml - - - Red - 80px - 100% - - - Green - - -``` - -> Entities without "extends": - -```xml - - - Red - 80px - 100% - - - Green - 80px - 100% - - -``` - -### Add a data entry - -__Use case__: Create an entity named `DivPanelGreen`, which is similar to the `DivPanel` entity, except that it has a specific panel color. - -> Entities with "extends": - -```xml - - - Red - 80px - 100% - - - #000000 - True - - -``` - -> Entities without "extends": - -```xml - - - Red - 80px - 100% - - - #000000 - 80px - 100% - True - - -``` - - -[test]: ./test.md -[data]: ./data.md -[action group]: ./test/action-groups.md -[actions]: ./test/actions.md diff --git a/docs/getting-started.md b/docs/getting-started.md deleted file mode 100644 index 87cb04f79..000000000 --- a/docs/getting-started.md +++ /dev/null @@ -1,368 +0,0 @@ -# Getting started - -
-[Find your version] of MFTF. -The latest Magento 2.3.x release supports MFTF 2.6.4. -The latest Magento 2.2.x release supports MFTF 2.5.3. -
- -## Prepare environment {#prepare-environment} - -Make sure that you have the following software installed and configured on your development environment: - -- [PHP version supported by the Magento instance under test][php] -- [Composer 1.3 or later][composer] -- [Java 1.8 or later][java] -- [Selenium Server Standalone 3.1 or later][selenium server] and [ChromeDriver 2.33 or later][chrome driver] or other webdriver in the same directory - -
-[PhpStorm] supports [Codeception test execution][], which is helpful when debugging. -
- -## Install Magento {#install-magento} - -Use instructions below to install Magento. - -### Step 1. Clone the `magento2` source code repository {#clone-magento} - -```bash -git clone https://github.com/magento/magento2.git -``` - -or - -```bash -git clone git@github.com:magento/magento2.git -``` - -### Step 2. Install dependencies {#install-dependencies} - -Checkout the Magento version that you are going to test. - -```bash -cd magento2/ -``` - -```bash -git checkout 2.4-develop -``` - -Install the Magento application. - -```bash -composer install -``` - -## Prepare Magento {#prepare-magento} - -Configure the following settings in Magento as described below. - -### WYSIWYG settings {#wysiwyg-settings} - -A Selenium web driver cannot enter data to fields with WYSIWYG. - -To disable the WYSIWYG and enable the web driver to process these fields as simple text areas: - -1. Log in to the Magento Admin as an administrator. -2. Navigate to **Stores** > **Settings** > **Configuration** > **General** > **Content Management**. -3. In the WYSIWYG Options section set the **Enable WYSIWYG Editor** option to **Disabled Completely**. -4. Click **Save Config**. - -or via command line: - -```bash -bin/magento config:set cms/wysiwyg/enabled disabled -``` - -Clean the cache after changing the configuration values: - -```bash -bin/magento cache:clean config full_page -``` - -
-When you want to test the WYSIWYG functionality, re-enable WYSIWYG in your test suite. -
- -### Security settings {#security-settings} - -To enable the **Admin Account Sharing** setting, to avoid unpredictable logout during a testing session, and disable the **Add Secret Key in URLs** setting, to open pages using direct URLs: - -1. Navigate to **Stores** > **Settings** > **Configuration** > **Advanced** > **Admin** > **Security**. -2. Set **Admin Account Sharing** to **Yes**. -3. Set **Add Secret Key to URLs** to **No**. -4. Click **Save Config**. - -or via command line: - -```bash -bin/magento config:set admin/security/admin_account_sharing 1 -``` - -```bash -bin/magento config:set admin/security/use_form_key 0 -``` - -Clean the cache after changing the configuration values: - -```bash -bin/magento cache:clean config full_page -``` - -### Testing with the Magento Two-Factor Authentication (2FA) extension {#2fa} - -If the Magento instance under test has the [Magento Two-Factor Authentication (2FA) extension][] installed and enabled, additional configurations is needed to run MFTF tests. Learn more in [Configure MFTF for Magento with Two-Factor Authentication (2FA)](./configure-2fa.md). - -### Webserver configuration {#web-server-configuration} - -MFTF does not support executing CLI commands if your web server points to `/pub` directory as recommended in the [Installation Guide][Installation Guide docroot]. For MFTF to execute the CLI commands, the web server must point to the Magento root directory. - -### Nginx settings {#nginx-settings} - -If the Nginx Web server is used on your development environment, then **Use Web Server Rewrites** setting in **Stores** > Settings > **Configuration** > **General** > **Web** > **Search Engine Optimization** must be set to **Yes**. - -Or via command line: - -```bash -bin/magento config:set web/seo/use_rewrites 1 -``` - -You must clean the cache after changing the configuration values: - -```bash -bin/magento cache:clean config full_page -``` - -To be able to run Magento command line commands in tests, add the following location block to the Nginx configuration file in the Magento root directory: - -```conf -location ~* ^/dev/tests/acceptance/utils($|/) { - root $MAGE_ROOT; - location ~ ^/dev/tests/acceptance/utils/command.php { - fastcgi_pass fastcgi_backend; - fastcgi_index index.php; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - include fastcgi_params; - } -} -``` - -## Set up an embedded MFTF {#setup-framework} - -This is the default setup of MFTF that you would need to cover your Magento project with functional tests. -It installs the framework using an existing Composer dependency such as `magento/magento2-functional-testing-framework`. -If you want to set up MFTF as a standalone tool, refer to [Set up a standalone MFTF][]. - -Install MFTF. - -```bash -composer install -``` - -### Step 1. Build the project {#build-project} - -In the Magento project root, run: - -```bash -vendor/bin/mftf build:project -``` - -If you use PhpStorm, generate a URN catalog: - -```bash -vendor/bin/mftf generate:urn-catalog .idea/misc.xml -``` - -If the file does not exist, add the `--force` option to create it: - -```bash -vendor/bin/mftf generate:urn-catalog --force .idea/misc.xml -``` - -See [`generate:urn-catalog`][] for more details. - -
-You can simplify command entry by adding the absolute path to the `vendor/bin` directory path to your PATH environment variable. -After adding the path, you can run `mftf` without having to include `vendor/bin`. -
- -### Step 2. Edit environmental settings {#environment-settings} - -In the `magento2/dev/tests/acceptance/` directory, edit the `.env` file to match your system. - -```bash -vim dev/tests/acceptance/.env -``` - -Specify the following parameters, which are required to launch tests: - -- `MAGENTO_BASE_URL` must contain a domain name of the Magento instance that will be tested. - Example: `MAGENTO_BASE_URL=http://magento.test` - -- `MAGENTO_BACKEND_NAME` must contain the relative path for the Admin area. - Example: `MAGENTO_BACKEND_NAME=admin` - -- `MAGENTO_ADMIN_USERNAME` must contain the username required for authorization in the Admin area. - Example: `MAGENTO_ADMIN_USERNAME=admin` - -- `MAGENTO_ADMIN_PASSWORD` must now be set up in the credentials file. See [Credentials Page][] for details. - -
-If the `MAGENTO_BASE_URL` contains a subdirectory like `http://magento.test/magento2ce`, specify `MAGENTO_CLI_COMMAND_PATH`. -
- -Learn more about environmental settings in [Configuration][]. - -### Step 3. Enable the Magento CLI commands - -In the Magento project root, run the following command to enable MFTF to send Magento CLI commands to your Magento instance. - - ```bash -cp dev/tests/acceptance/.htaccess.sample dev/tests/acceptance/.htaccess -``` - -### Step 4. Generate and run tests {#run-tests} - -To run tests, you need a running Selenium server and [`mftf`][] commands. - -#### Run the Selenium server {#selenium-server} - -Run the Selenium server in the terminal. -For example, the following commands download and run the Selenium server for Google Chrome: - -```bash -curl -O http://selenium-release.storage.googleapis.com/3.14/selenium-server-standalone-3.14.0.jar -``` - -```bash -java -Dwebdriver.chrome.driver=chromedriver -jar selenium-server-standalone-3.14.0.jar -``` - -#### Generate and run all tests {#run-all-tests} - -```bash -vendor/bin/mftf generate:tests -``` - -```bash -vendor/bin/codecept run functional -c dev/tests/acceptance/codeception.yml -``` - -See more commands in [`codecept`][]. - -#### Run a simple test {#run-test} - -To clean up the previously generated tests, and then generate and run a single test `AdminLoginSuccessfulTest`, run: - -```bash -vendor/bin/mftf run:test AdminLoginSuccessfulTest --remove -``` - -See more commands in [`mftf`][]. - -### Step 5. Generate reports {#reports} - -During testing, MFTF generates test reports in CLI. You can generate visual representations of the report data using the [Allure Framework][]. To view the reports in a GUI: - -- [Install Allure][] -- Run the tool to serve the artifacts in `dev/tests/acceptance/tests/_output/allure-results/`: - -```bash -allure serve dev/tests/acceptance/tests/_output/allure-results/ -``` - -Learn more about Allure in the [official documentation][allure docs]. - -## Set up a standalone MFTF - -MFTF is a root level Magento dependency, but it is also available for use as a standalone application. You may want to use a standalone application when you develop for or contribute to MFTF, which facilitates debugging and tracking changes. These guidelines demonstrate how to set up and run Magento acceptance functional tests using standalone MFTF. - -### Prerequisites - -This installation requires a local instance of the Magento application. -MFTF uses the [tests from Magento modules][mftf tests] as well as the `app/autoload.php` file. - -### Step 1. Clone the MFTF repository - -If you develop or contribute to MFTF, it makes sense to clone your fork of the MFTF repository. -For contribution guidelines, refer to the [Contribution Guidelines for the Magento Functional Testing Framework][contributing]. - -### Step 2. Install the MFTF - -```bash -cd magento2-functional-testing-framework -``` - -```bash -composer install -``` - -### Step 3. Build the project - -```bash -bin/mftf build:project -``` - -### Step 4. Edit environment settings - -In the `dev/.env` file, define the [basic configuration][] and [`MAGENTO_BP`][] parameters. - -### Step 5. Enable the Magento CLI commands {#add-cli-commands} - -Copy the `etc/config/command.php` file into your Magento installation at `/dev/tests/acceptance/utils/`. -Create the `utils/` directory, if you didn't find it. - -### Step 6. Remove the MFTF package dependency in Magento - -MFTF uses the Magento `app/autoload.php` file to read Magento modules. -The MFTF dependency in Magento supersedes the standalone registered namespaces unless it is removed at a Composer level. - -```bash -composer remove magento/magento2-functional-testing-framework --dev -d -``` - -### Step 7. Run a simple test - -Generate and run a single test that will check your logging to the Magento Admin functionality: - -```bash -bin/mftf run:test AdminLoginSuccessfulTest -``` - -You can find the generated test at `dev/tests/functional/tests/MFTF/_generated/default/`. - -### Step 8. Generate Allure reports - -The standalone MFTF generates Allure reports at `dev/tests/_output/allure-results/`. -Run the Allure server pointing to this directory: - -```bash -allure serve dev/tests/_output/allure-results/ -``` - - - -[`codecept`]: commands/codeception.html -[`generate:urn-catalog`]: commands/mftf.html#generateurn-catalog -[`MAGENTO_BP`]: configuration.html#magento_bp -[`mftf`]: commands/mftf.html -[allure docs]: https://docs.qameta.io/allure/ -[Allure Framework]: https://github.com/allure-framework -[basic configuration]: configuration.html#basic-configuration -[chrome driver]: https://sites.google.com/a/chromium.org/chromedriver/downloads -[Codeception Test execution]: https://blog.jetbrains.com/phpstorm/2017/03/codeception-support-comes-to-phpstorm-2017-1/ -[composer]: https://getcomposer.org/download/ -[Configuration]: configuration.html -[contributing]: https://github.com/magento/magento2-functional-testing-framework/blob/develop/.github/CONTRIBUTING.md -[install Allure]: https://github.com/allure-framework/allure2#download -[java]: https://www.oracle.com/java/technologies/downloads/ -[mftf tests]: introduction.html#mftf-tests -[php]: https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements.html -[PhpStorm]: https://www.jetbrains.com/phpstorm/ -[selenium server]: https://www.seleniumhq.org/download/ -[Set up a standalone MFTF]: #set-up-a-standalone-mftf -[test suite]: suite.html -[Find your version]: introduction.html#find-your-mftf-version -[Installation Guide docroot]: https://devdocs.magento.com/guides/v2.4/install-gde/tutorials/change-docroot-to-pub.html -[Magento Two-Factor Authentication (2FA) extension]: https://devdocs.magento.com/guides/v2.4/security/two-factor-authentication.html -[Credentials Page]: https://devdocs.magento.com/mftf/docs/credentials.html diff --git a/docs/guides/action-groups.md b/docs/guides/action-groups.md deleted file mode 100644 index 30c531e4e..000000000 --- a/docs/guides/action-groups.md +++ /dev/null @@ -1,82 +0,0 @@ -# Action Group Best Practices - -We strive to write tests using only action groups. Fortunately, we have built up a large set of action groups to get started. We can make use of them and extend them for our own specific needs. In some cases, we may never even need to write action groups of our own. We may be able to simply chain together calls to existing action groups to implement our new test case. - -## Why use Action Groups? - -Action groups simplify maintainability by reducing duplication. Because they are re-usable building blocks, odds are that they are already made use of by existing tests in the Magento codebase. This proves their stability through real-world use. Take for example, the action group named `LoginAsAdmin`: - -```xml - - - Login to Backend Admin using provided User Data. PLEASE NOTE: This Action Group does NOT validate that you are Logged In. - - - - - - - - - - - -``` - -Logging in to the admin panel is one of the most used action groups. It is used around 1,500 times at the time of this writing. - -Imagine if this was not an action group and instead we were to copy and paste these 5 actions every time. In that scenario, if a small change was needed, it would require a lot of work. But with the action group, we can make the change in one place. - -## How to extend action groups - -Again using `LoginAsAdmin` as our example, we trim away metadata to clearly reveal that this action group performs 5 actions: - -```xml - - ... - - - - - - -``` - -This works against the standard Magento admin panel login page. Bu imagine we are working on a Magento extension that adds a CAPTCHA field to the login page. If we create and activate this extension and then run all existing tests, we can expect almost everything to fail because the CAPTCHA field is left unfilled. - -We can overcome this by making use of MFTF's extensibility. All we need to do is to provide a "merge" that modifies the existing `LoginAsAdmin` action group. Our merge file will look like: - -```xml - - - -``` - -Because the name of this merge is also `LoginAsAdmin`, the two get merged together and an additional step happens everytime this action group is used. - -To continue this example, imagine someone else is working on a 'Two-Factor Authentication' extension and they also provide a merge for the `LoginAsAdmin` action group. Their merge looks similar to what we have already seen. The only difference is that this time we fill a different field: - -```xml - - - -``` - -Bringing it all together, our resulting `LoginAsAdmin` action group becomes this: - -```xml - - ... - - - - - - - - -``` - -No one file contains this exact content as above, but instead all three files come together to form this action group. - -This extensibility can be applied in many ways. We can use it to affect existing Magento entities such as tests, action groups, and data. Not so obvious is that this tehcnique can be used within your own entities to make them more maintainable as well. diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md deleted file mode 100644 index 80cf134d8..000000000 --- a/docs/guides/cicd.md +++ /dev/null @@ -1,114 +0,0 @@ -# How to use MFTF in CICD - -To integrate MFTF tests into your CICD pipeline, it is best to start with the conceptual flow of the pipeline code. - -## Concept - -The overall workflow that tests should follow is: - -- Obtain a Magento instance + install pre-requisites. -- Generate the tests. - - Set options for single or parallel running. -- Delegate and run tests and gather test-run artifacts. - - Re-run options. -- Generate the Allure reports from the results. - -## Obtain a Magento instance - -To start, we need a Magento instance to operate against for test generation and execution. - -```bash -git clone https://github.com/magento/magento2 -``` - -or - -```bash -composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce -``` - -For more information on installing magento see [Install Magento using Composer][]. - -After installing the Magento instance, set a couple of configurations to the Magento instance: - -```bash -bin/magento config:set general/locale/timezone America/Los_Angeles -bin/magento config:set admin/security/admin_account_sharing 1 -bin/magento config:set admin/security/use_form_key 0 -bin/magento config:set cms/wysiwyg/enabled disabled -``` - -These set the default state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step to do it. - -If your magento instance has Two-Factor Authentication enabled, see [Configure 2FA][] to configure MFTF tests. - -## Install Allure - -This is required for generating the report after your test runs. See [Allure][] for details. - -## Generate tests - -### Single execution - -Generate tests based on what you want to run: - -```bash -vendor/bin/mftf generate:tests -``` - -This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt`. - -### Parallel execution - -To generate all tests for use in parallel nodes: - -```bash -vendor/bin/mftf generate:tests --config parallel -``` - -This generates a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. - -## Delegate and run tests - -### Single execution - -If you are running on a single node, call: - -```bash -vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt -``` - -### Parallel execution - -You can optimize your pipeline by running tests in parallel across multiple nodes. - -Tests can be split up into roughly equal running groups using `--config parallel`. - -You do not want to perform installations on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. - -The groups can be then distributed on each of the nodes and run separately in an isolated environment. - -- Stash artifacts from main node and un-stash on current node. -- Run `vendor/bin/mftf run:manifest ` on current node. -- Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. - -### Rerun options - -In either single or parallel execution, to re-run failed tests, simply add the `run:failed` command after executing a manifest: - -```bash -vendor/bin/mftf run:failed -``` - -### Generate Allure report - -In the main node, generate reports using your `` into a desired output path: - -```bash -allure generate -c -o -``` - - -[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.4/install-gde/composer.html -[Configure 2FA]: ../configure-2fa.md -[Allure]: https://docs.qameta.io/allure/ diff --git a/docs/guides/git-vs-composer-install.md b/docs/guides/git-vs-composer-install.md deleted file mode 100644 index fd9006cc1..000000000 --- a/docs/guides/git-vs-composer-install.md +++ /dev/null @@ -1,83 +0,0 @@ -# Git vs Composer installation of Magento with MFTF - -Depending on how you plan to use Magnto code, there are different options for installing Magento. - -## GitHub Installation - -If you are contributing a pull request to the Magento 2 codebase, download Magento 2 from our GitHub repository. Contribution to the codebase is done using the 'fork and pull' model where contributors maintain their own fork of the repo. This repo is then used to submit a pull request to the base repo. - -Install guide: [GitHub Installation][] - -## Composer based Installation - -A Composer install downloads released packages of Magento 2 from the composer repo [https://repo.magento.com](https://repo.magento.com). - -All Magento modules and their MFTF tests are put under `` directory, for convenience of 3rd party developers. With this setup, you can keep your custom modules separate from core modules. You can also develop modules in a separate VCS repository and add them to your `composer.json` which installs them into the `vendor` directory. - -Install guide: [Composer based Installation][] - -## MFTF Installation - -After installing your Magento project in either of the above ways, the composer dependency `magento/magento2-functional-testing-framework` downloads and installs MFTF. MFTF is embedded in your Magento 2 installation and will cover your project with functional tests. - -If you want to contribute a pull request into MFTF codebase, you will need to install MFTF in the [Standalone][] mode. - -## Managing modules - Composer vs GitHub - -### Via GitHub - -Cloning the Magento 2 git repository is a way of installing where you do not have to worry about matching your codebase with production. Your version control system generally holds and manages your `app/code` folder and you can do manual, ad-hoc development here. - -### Via Composer - -Magento advocates the use of composer for managing modules. When you install a module through composer, it is added to `vendor//`. - -When developing your own module or adding MFTF tests to a module, you should not edit in `vendor` because a composer update could overwrite your changes. Instead, overwrite a module under `vendor` by adding files or cloning your module-specific Git repo to `app/code//`. - -To distribute the module and its tests, you can initialize a git repo and create a [composer package][]. In this way others will be able to download and install your module and access your tests as a composer package, in their `` folder. - -## MFTF test materials location - -- For GitHub installations, MFTF test materials are located in `/app/code///Test/Mftf/`. This is the directory for new tests or to maintain existing ones. -- For Composer-based installations, MFTF test materials are located at `/vendor///Test/Mftf/`. This is the directory to run tests fetched by Composer. - -The file structure under both paths is the same: - -```tree - -├── ActionGroup -│   └── ... -├── Data -│   └── ... -├── Metadata -│   └── ... -├── Page -│   └── ... -├── Section -│   └── ... -├── Suite -│   └── ... -└── Test - └── ... -``` - -## How ModuleResolver reads modules - -With either type of installation, all tests and test data are read and merged by MFTF's ModuleResolver in this order: - -1. `/app/code///Test/Mftf/` -1. `/vendor///Test/Mftf/` -1. `/dev/tests/acceptance/tests/functional///` - -## Conclusion - -There is no difference between having the test materials in `app/code` or in `/vendor`: it works the same. Composer-based installs may benefit teams when there is a need to match file systems in `development` and `production`. - -If you are a contributing developer with an understanding of Git and Composer commands, you can choose the GitHub installation method instead. - - - -[Composer based Installation]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html -[GitHub Installation]: https://devdocs.magento.com/guides/v2.3/install-gde/prereq/dev_install.html -[Standalone]: ../getting-started.html#set-up-a-standalone-mftf -[composer package]: https://devdocs.magento.com/guides/v2.3/extension-dev-guide/package/package_module.html diff --git a/docs/guides/selectors.md b/docs/guides/selectors.md deleted file mode 100644 index d1441865a..000000000 --- a/docs/guides/selectors.md +++ /dev/null @@ -1,346 +0,0 @@ -# How To write good selectors - -Selectors are the atomic unit of test writing. They fit into the hierarchy like this: MFTF tests make use of action groups > which are made up of actions > which interact with page objects > which contain elements > which are specified by selectors. Because they are fundamental building blocks, we must take care when writing them. - -## What is a selector? - -A "selector" works like an address to an element in the Document Object Model (DOM). It specifies page elements and allows MFTF to interact with them. -By 'element' we mean things such as input fields, buttons, tables, divs, etc. -By 'interact' we mean actions such as click, fill field, etc. - -Selectors live inside of MFTF page objects and are meant to be highly re-usable amongst all tests. They can be written in either CSS or XPath. - -## Why are good selectors important? - -Good selectors are important because they are the most re-used component of functional testing. They are the lowest building blocks of tests; the foundation. If they are unstable then everything else built on top of them will inherit that instability. - -## How do I write good selectors? - -We could cover this subject with an infinite amount of documentation and some lessons only come from experience. This guide explains some DOs and DONTs to help you along the way towards selector mastery. - -### Inspecting the DOM - -To write a selector you need to be able to see the DOM and find the element within it. Fortunately you do not have to look at the entire DOM every time. Nor do you have to read it from top to bottom. Instead you can make use of your browsers built-in developer tools or go a step further and try out some popular browser extensions. - -See these links for more information about built-in browser developer tools: - -* [Chrome Developer Tools](https://developers.google.com/web/tools/chrome-devtools/) -* [Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools) - -See these links for common browser addons that may offer advantages over browser developer tools: - -* [Live editor for CSS, Less & Sass - Magic CSS](https://chrome.google.com/webstore/detail/live-editor-for-css-less/ifhikkcafabcgolfjegfcgloomalapol?hl=en) -* [XPath Helper](https://chrome.google.com/webstore/detail/xpath-helper/hgimnogjllphhhkhlmebbmlgjoejdpjl?hl=en) - -### CSS vs XPath - -There are similarities and differences between CSS and XPath. Both are powerful and complex in ways that are outside of the scope of this document. -In general: - -* CSS is more stable, easier to read, and easier to maintain (typically). -* XPath provides several powerful tools and it has been around the longest so it is well documented. -* XPath can be less stable and potentially unsupported by certain actions in Selenium. - -### Priority - -The best and most simple selector will always be to use an element ID: `#some-id-here`. If only we were so lucky to have this every time. - -When writing selectors, you should prioritize finding in this order: - -1. ID, name, class, or anything else that is unique to the element -2. Complex CSS selectors -3. XPath selectors -4. If none of the above work for you, then the last resort is to ask a developer to add a unique ID or class to the element you are trying to select. - -We suggest the use of CSS selectors above XPath selectors when possible. - -### Writing proper selectors - -There are correct ways of writing selectors and incorrect ways. These suggestions will help you write better selectors. - -#### Incorrect - copy selector/xpath - -DO NOT right click on an element in your browser developer tools and select "Copy selector" or "Copy XPath" and simply use that as your selector. These auto-generated selectors are prime examples of what not to do. - -These are bad: - -```css -#html-body > section > div > div > div > div -``` - -```xpath -//*[@id='html-body']/section/div/div/div/div -``` - -Both include unnecessary hierarchical details. As written, we are looking for a `div` inside of a `div` inside of a `div` inside of... you get the picture. If an application developer adds another `div` parent tomorrow, for whatever reason, this selector will break. Furthermore, when reading it, it is not clear what the intended target is. It may also grab other elements that were not intended. - -#### Do not be too general - -DO NOT make your selectors too generic. If a selector is too generic, there is a high probability that it will match multiple elements on the page. Maybe not today, but perhaps tomorrow when the application being tested changes. - -These are bad: - -```html -input[name*='firstname'] -``` - -The `*=` means `contains`. The selector is saying 'find an input whose name contains the string "firstname"'. But if a future change adds a new element to the page whose name also contains "firstname", then this selector will match two elements and that is bad. - -```css -.add -``` - -Similarly here, this will match all elements which contains the class `.add`. This is brittle and susceptible to breaking when new elements/styles are added to the page. - -#### Avoid being too specific - -DO NOT make your selectors too specific either. If a selector is too specific, there is a high probability that it will break due to even minor changes to the application being tested. - -These are bad: - -```css -#container .dashboard-advanced-reports .dashboard-advanced-reports-description .dashboard-advanced-reports-title -``` - -This selector is too brittle. It would break very easily if an application developer does something as simple as adding a parent container for style reasons. - -```xpath -//*[@id='container']/*[@class='dashboard-advanced-reports']/*[@class='dashboard-advanced-reports-description']/*[@class='dashboard-advanced-reports-title'] -``` - -This is the same selector as above, but represented in XPath instead of CSS. It is brittle for the same reasons. - -#### XPath selectors should not use @attribute="foo" - -This XPath is fragile. It would fail if the attribute was `attribute="foo bar"`. Instead you should use `contains(@attribute, "foo")` where @attribute is any valid attribute such as @text or @class. - -#### CSS and XPath selectors should avoid making use of hardcoded indices - -Hardcoded values are by definition not flexible. A hardcoded index may change if new code is introduced. Instead, parameterize the selector. - -GOOD: .foo:nth-of-type({{index}}) - -BAD: .foo:nth-of-type(1) - -GOOD: button[contains(@id, "foo")][{{index}}] - -BAD: button[contains(@id, "foo")][1] - -GOOD: #actions__{{index}}__aggregator - -BAD: #actions__1__aggregator - -#### CSS and XPath selectors MUST NOT reference the @data-bind attribute - -The @data-bind attribute is used by KnockoutJS, a framework Magento uses to create dynamic Javascript pages. Since this @data-bind attribute is tied to a specific framework, it should not be used for selectors. If Magento decides to use a different framework then these @data-bind selectors would break. - -#### Use isolation - -You should think in terms of "isolation" when writing new selectors. - -For example, say you have a login form that contains a username field, a password field, and a 'Sign In' button. First isolate the parent element. Perhaps it's `#login-form`. Then target the child element under that parent element: `.sign-in-button` The result is `#login-form .sign-in-button`. - -Using isolation techniques reduces the amount of DOM that needs to be processed. This makes the selector both accurate and efficient. - -#### Use advanced notation - -If you need to interact with the parent element but it is too generic, and the internal contents are unique then you need to: - -1. Target the unique internal contents first. -1. Then jump to the parent element using `::parent`. - -Imagine you want to find a table row that contains the string "Jerry Seinfeld". You can use the following XPath selector: - -```xpath -//div[contains(text(), 'Jerry Seinfeld')]/parent::td/parent::tr -``` - -Note in this instance that CSS does not have an equivalent to `::parent`, so XPath is a better choice. - -### CSS Examples - -Examples of common HTML elements and the corresponding selector to find that element in the DOM: - -Type|HTML|Selector ----|---|--- -IDs|`
`|`#idname` -Classes|`
`|`.classname` -HTML Tags|`
`|`div` -HTML Tag & ID|`
`|`div#idname` -HTML Tag & Class|`
`|`div.classname` -ID & Class|`
`|`#idname.classname` -HTML Tag & ID & Class|`
`|`div#idname.classname` - -Examples of common CSS selector operators and their purpose: - -Symbol|Name|Purpose|Selector ----|---|---|--- -`*`|Universal Selector|Allows you to select ALL ELEMENTS on the Page. Wild Card.|`*` -Whitespace|Descendant Combinator|Allows you to combine 2 or more selectors.|`#idname .classname` -`>`|Child Combinator|Allows you to select the top-level elements THAT FOLLOWS another specified element.|`#idname > .classname` -`+`|Adjacent Sibling Combinator|Allows you to select an element THAT FOLLOWS DIRECTLY AFTER another specified element.|`#idname + .classname` -`~`|General Sibling Combinator|Allows you to select an element THAT FOLLOWS (directly or indirectly) another specified element.|`#idname ~ .classname` - -Examples of CSS attribute operators and their purpose: - -Symbol|Purpose|Example ----|---|--- -`=`|Returns all elements that CONTAIN the EXACT string in the value.|`[attribute='value']` -`*=`|Returns all elements that CONTAINS the substring in the value.|`[attribute*='value']` -`~=`|Returns all elements that CONTAINS the given words delimited by spaces in the value.|`[attribute~='value']` -`$=`|Returns all elements that ENDS WITH the substring in the value.|`[attribute$='value']` -`^=`|Returns all elements that BEGIN EXACTLY WITH the substring in the value.|`[attribute^='value']` -`!=`|Returns all elements that either DOES NOT HAVE the given attribute or the value of the attribute is NOT EQUAL to the value.|`[attribute!='value']` - -### XPath Examples - -#### `/` vs `//` - -The absolute XPath selector is a single forward slash `/`. It is used to provide a direct path to the element from the root element. - -WARNING: The `/` selector is brittle and should be used sparingly. - -Here is an example of what NOT to do, but this demonstrates how the selector works: - -```xpath -/html/body/div[2]/div/div[2]/div[1]/div[2]/form/div/input -``` - -In the BAD example above, we are specifying a very precise path to an input element in the DOM, starting from the very top of the document. - -Similarly, the relative XPath selector is a double forward slash `//`. It is used to start searching for an element anywhere in the DOM starting from the specified element. If no element is defined, the entire DOM is searched. - -Example: - -```xpath -//div[@class=’form-group’]//input[@id='user-message'] -``` - -In the `GOOD` example above, all `
` elements in the DOM are matched first. Then all `` with `
` as one of its parents are matched. The parent does not have to immediately precede it since it uses another double forward slash `//`. - -#### Parent Selectors - -The parent selector (`..`) allows you to jump to the parent element. - -Example #1: - -Given this HTML: - -```html - - -
Unique Value
- - -``` - -We can locate the `` element with this selector: - -```xpath -//*[text()='Unique Value']/../.. -``` - -Example #2: - -Given this HTML: - -```html - - - Edit - - -
Unique Value
- - -``` - -We can locate the `` element with this selector: - -```xpath -//div[text()='Unique Value']/../..//a -``` - -#### Attribute Selectors - -Attribute selectors allow you to select elements that match a specific attribute value. - -Examples: - -Attribute|HTML|Selector ----|---|--- -id|`
`|`//*[@id='idname']` -class|`
`|`//*[@class='classname']` -type|`