Skip to content

Scraping Data ​

Extract specific data from web page elements into your .xlsx report or export directly as structured output files.


Overview ​

The Scraping Data field type targets individual DOM nodes to extract text, HTML attributes, form values, or selected dropdown options. Extracted values are saved directly under the field's column header in Scraper Data and can be referenced dynamically by subsequent fields using {$fieldName$} variable syntax.


Configuration Options ​

OptionDescriptionRequired
Selector TypeSyntax type (CSS Selector or XPath).Yes
Selector QueryTarget element selector query.Yes
Which Element Option ScrapingTarget property to extract from the element.Yes
Wait until element is found in the pagePauses execution until the element mounts in the DOM.No
Are you using custom javascript function for return field responses?Enables custom post-processing via a JavaScript listener.No
Download a file with scraped data?Saves extracted data directly as a downloadable file.No
File TypeOutput format when downloading a file (Text File, CSV File, JSON File).When download enabled
Wait, the unit file download is complete?Pauses automation until the browser download finishes.No
Do you want to set the custom name of the file?Enables naming the exported download file.No
Enter file nameCustom filename string (without extension).When custom name enabled

Element Scraping Targets ​

PropertyExtraction TargetHTML Example
Element TextVisible text (innerText).<div class="tag">Active</div> → Active
Element Text ContentRaw text including hidden descendants (textContent).<div style="display:none">123</div> → 123
Element ValueInput or textarea value.<input value="USR-99" /> → USR-99
Element HTMLInner markup (innerHTML).<p><strong>Bold</strong></p> → <strong>Bold</strong>
Element AttributeValue of any specific attribute (e.g., href, data-id).<a href="/doc.pdf"> → /doc.pdf
Element Selected Option TextVisible label of active <option> in a <select>.<option selected>California</option> → California
Element Selected Option ValueValue attribute of active <option> in a <select>.<option value="CA" selected> → CA

Custom JavaScript Processing ​

Transform or sanitize scraped values before committing them:

  1. Add a JavaScript Code field positioned directly above this field.
  2. Enable Are you using custom javascript function for return field responses?.
  3. Register a listener:
javascript
$fns.field.listener('EDF-FIELD-SCRAPING-DATA', (output, callback) => {
  console.log('Raw Scraped Data:', output.response);

  // Normalize string to uppercase and trim whitespace
  const sanitized = output.response.trim().toUpperCase();

  // Return cleaned value
  callback({ status: true, message: 'DONE', data: sanitized });
});

$fns.return('1');

Practical Examples ​

Example 1: Scrape Order Confirmation Number ​

text
Selector Type: CSS Selector
Selector Query: span#order-confirmation-number
Which Element Option: Element Text
Wait until element is found: Enabled

Example 2: Extract Selected Dropdown Value ​

text
Selector Type: CSS Selector
Selector Query: select#billing-state
Which Element Option: Element Selected Option Value
text
Selector Type: CSS Selector
Selector Query: .avatar-profile img
Which Element Option: Element Attribute
Element Attribute Name: src

Best Practices ​

  • Wait for Dynamic Elements: Always enable Wait until element is found in the page when scraping elements hydrated asynchronously by client-side frameworks.
  • Reference as Variables: Use {$fieldName$} in subsequent form steps to input the scraped value into downstream forms.
  • Use Stable Selectors: Inspect elements with Alt + Shift + I to ensure selectors avoid dynamic ephemeral class names.