It’s never been easier to put together top-notch digital signage with software like ScreenCloud. But if you want to go beyond the basic news, weather and traffic information, coding new dashboards or copying and pasting information from around the web onto your signage can get tedious and time-consuming.

The better way is to use data scraping to gather data at scale for a fraction of the effort. You’ll make your screens more engaging, whether they’re for customers in a waiting room or for your team’s break room. Gather the top trending keywords, live sales data, and customer support team progress to keep everyone’s finger on the pulse, without adding more hours to your work week. Here’s how, with ScreenCloud plus popular scraping tools.

Scraping web content is copying data from a web page and storing it for later use. You can manually scrape data by copying and pasting, but that’s tedious and time consuming. The better way is to hire a robot and get scraping software to copy data for you. It’s faster and more accurate—and can automatically re-scrape data anytime anything changes, without any extra work on your part.

Typically, you’ll first open a website in the scraping tool, select what you want to extract, and then have the software export it in a CSV file or push it to an API or Webhooks URL to use elsewhere. You can then take your scraped data and use it anywhere.

Data scraping—in a way—is how Google Search was built, how ChatGPT’s AI was trained, and how many companies leverage business intelligence to gather data about competitors to improve decision-making. And it’s just as useful for grabbing the latest headlines and team performance numbers for your company’s signage. It’s workplace automation that can work for your team.

There are four main methods to scrape data from websites:

Using Google Sheets or Excel to scrape HTML/XML data

Scraping data via API (if the target website has one)

Coding your own data scraper using Python

Here’s how to use each of those—and how to get your scraped data onto your ScreenCloud signage.

Let’s start with the easiest way to scrape data from a website: A web scraping app. These tools help you extract data without having to code or understand how websites work. To set  up a scraping job, you’ll add a URL, select the data you want to collect, and either save it to a CSV file or push it to another app once the data is collected. Some of these tools employ AI to get around tricky websites, making them more adaptable to website changes over time.

<center>ScreenCloud Dashboards can copy a widget or section from any site, and put it on your signage.</center>

ScreenCloud includes a basic scraping app built for digital signage: ScreenCloud Dashboards. It’s perfect to capture a portion of a webpage—a graph from Tableau or PowerBI, a table from Jira or Trello, a dashboard from Notion or Airtable—and display it on your signage. And it works much like other scraping tools. You enter a website address, select the bit of the app you’d like to copy, and ScreenCloud will show a live, updated version of it on your signage.

That works great when you want to show an entire widget, graph, or section of a site or app. If you only want a more specific bit of data—a single value, a headline sentence, or a table’s worth of data to sort and filter—you’ll need a dedicated data scraping app. Here’s a shortlist of 3 solid apps you can use for data scraping:

Octoparse is the robust choice, trusted by enterprises—and starts at $75/month. It can run on your local computer or in the cloud, with a no-code interface that lets you select the data you want to extract, and workflows to collect everything you need from a site. You can see the expected end result as a table at the bottom of the screen, so you don’t have to run multiple attempts to get what you want. It has a learning curve due to its flexibility and advanced scraping settings—but its advanced tools means it can solve CAPATCHAs for you and keep your scraper from getting blocked. And it has integrations and Webhook support to send your scraped data to your signage..

Browse.ai is more beginner-friendly, offering ways to monitor URLs and scrape any updates with a $19/month Starter plan. It lets you configure robots that can access dynamic list pages to extract product lists or hotel pricing, for example. The data can then be sent to Google Sheets for an easy way to display it on your signage. 

Webscraper.io is the free option, at least if you’re scraping locally using your own browser. Install its browser extension and start scraping data right away. The data is stored in a CSV, XLSX or JSON file, ready to upload to your favorite spreadsheet or database app to then pass along to your digital signage. For cloud automation, advanced scraping features and API access, the pricing starts at $50 per month.

<center>Load a site in Browse.ai, and its bot will let you pick any bits of text from the website to monitor and copy automatically.</center>

Let’s set up an automation with Browse.ai to see how it works with ScreenCloud signage. Create an account, and select to monitor a site for changes (you could also have it extract data from a table, if you want). Then enter the website you want to monitor, add Browse.ai’s Chrome extension if you haven’t already, then click Start Recording.

Browse.ai will load your site with a robot icon in the top right. Sign in as normal, find the page you need, then click the robot, choose to select text, and pick the item on the site you want to monitor.

<center>Set a schedule in Browse.ai to check your site or app for changes.</center>

Finally, save and set up a schedule, and every time Browse.ai notices a change in that text on the site it’s monitoring, it’ll let you know. And you can do more from the Integrations tab. There, you can have Browse.AI push the updated data to another app via API or Webhooks, get it to save data to Google Sheets, or send it to automation platforms like Zapier.

<center>Use ScreenCloud Playgrounds to build an auto-updating dashboard with your scraped data</center>

That same general workflow works with all of the scraping apps recommended. Whenever your app scrapes something new, it can send it to Zapier—which then can push it along to your newly auto-updating ScreenCloud dashboard.

Option 2: Use Excel or Google Sheets to scrape HTML data

Spreadsheets were the original killer app on the earliest personal computers—and it’s surprising how often spreadsheets remain the best tool for so many tasks after all these years. They’ve gained more advanced functions over the years, including tools to scrape HTML or XML data from a website using the IMPORT formula family. Add URLs to one column, the formula to another, and you’ll get your data scraped in minutes.

This is a free, simple way to scrape easy-to-access data from sites that don’t require a login. More advanced workflows will still need a full scraping tool, but for the more basic tasks spreadsheets can cover the basics. And once you’ve scraped data with a spreadsheet, you can format it then display it on ScreenCloud with the Google Sheets integration or an Excel Online embed.

Here are how to use the core spreadsheet web scraping functions:

<center>A Wikipedia table, parsed into Google Sheets in seconds</center>

Use IMPORTHTML when you need to extract a list or table from a website’s HTML:

=IMPORTHTML(url, query, index)

Start by adding the URL of the target page. Be sure to enclose it in quotation marks.

Then, fill out the query. This can only be one of two options: use “table” for extracting data from tables and “lists” for, you guessed it, extracting lists. Enclose this in quotation marks too.

Finally, the index refers to which table or list you want. If the page has more than one table, count starting from the top left and write that number in the formula. If you don’t get the desired table, increase or decrease that number until you get the right one.

Here’s an example where we’re extracting a list of all documented helicopter escape attempts from prisons. Open the link on your browser to see the page, and then paste the formula below in Google Sheets to see what happens:

=IMPORTHTML("https://en.wikipedia.org/wiki/List_of_helicopter_prison_escapes", "table", 2)

The list on the Wikipedia page should now be displayed inside your spreadsheet.

Be sure to include http:// or https:// at the beginning of every URL.

Don’t forget the quotation marks for the URL and query.

Separate each argument (url, query, index) using a comma.

In Google Sheets, make sure there’s space below and in front of the cell where you’re writing the formula for best results.

<center>Copy every URL from a website in a second in Google Sheets</center>

The IMPORTXML formula is more versatile but slightly harder to use. Despite its name, you can use it to extract data not only from XML files but also from HTML and CSV files—which means you can copy data from almost any website. To use it, write the following:

=IMPORTXML(url, xpath_query)

Start by filling the URL argument with the target page. Then, fill in the xpath_query with the identifier of the data you want to extract. You have to input a path to the data you want, like how you input a path in your file explorer to reach a folder inside your computer. Here’s the syntax guide to help you put together your query.

To discover the path to the data you want to extract on the web page:

Open your browser and navigate to the page you’re scraping.

On your keyboard, use the CTRL + SHIFT + C (CMD + SHIFT + C on a Mac) keyboard shortcut to open the developer tools and start the element inspector. Alternatively, you can right-click the content and then click on Inspect.

Click on the content that you want to extract.

Notice that the page’s HTML on the developer tools pane highlights the piece of code that’s displaying that element. This is what we’re looking for. Write down the HTML tag that’s marking the content you want to extract on the xpath_query argument. Consider also noting down any IDs and classes within that tag, as these can be used to further identify the data that you want to extract.

Here’s an example, where we’re extracting all the links from the list of micronations from Wikipedia:

=IMPORTXML("https://en.wikipedia.org/wiki/List_of_micronations", "//@href")

Remember to add quotation marks around the URL and the XPath query

Build your own query by exploring the XPath query syntax in this WC3 guide.

<center>Google Sheets can scrape RSS and Podcast feeds, too.</center>

Remember RSS feeds? They’re great to keep up with updates on a website, and are still the only way to publish podcasts. You can import RSS feeds and their latest information using this formula in Google Sheets:

=IMPORTFEED(url, query, headers, number_of_items)

This formula only needs the URL to work. All the other arguments are optional. Here’s what they do:

query lets you select what information you want to grab, for example just the name or the date

headers can be set to TRUE to grab the header row of the feed, labeling columns appropriately

number_of_items lets you select how many rows you want to retrieve. Adding 3 as an argument would retrieve the 3 first rows of the feed.

Here’s the New York Times’ The Daily podcast RSS feed, grabbing the summary of each of the first 5 most recent episodes:

=IMPORTFEED("https://feeds.simplecast.com/54nAGcIl", "items summary", TRUE, 5)

Don’t forget quotes around the URL and query.

The headers and number arguments don’t need quotes around them.

If you want to set the number of items but don’t want to enter a query or get the headers, remember to fill each argument with blanks by using a double set of empty quotation marks. For example, =IMPORTFEED("https://feeds.simplecast.com/54nAGcIl", "", "", 1) will retrieve only the first row of the feed with an empty query and no header.

And, pro-tip: ScreenCloud supports RSS feeds natively, for another way to get live-updated data on your signage without needing to scrape it first.

In some cases, the website you’re visiting may offer an API: an application programming interface. APIs can be used to build third-party services on top of the data and functionality of an app or website. For example, by accessing the Facebook API, you could build a social media scheduling app by pushing and pulling data to and from the platform. And, since APIs let you read data from a platform or website, you can use them to scrape data.

The upside of this method is that the end result is much more accurate than other options on the list. You’ll be accessing data sourced directly from the platform’s databases. There are some drawbacks, though:

It’s up to the website’s owners to decide what kinds of data they’ll offer via API, so there may be cases where you can’t access all the data you need. There’s also the possibility that some API endpoints require you to create an account or pay a subscription to access.

The data output is formatted in JSON (JavaScript Object Notation), a format that makes data easier to read for machines (and somewhat challenging for humans). You’ll have to convert the JSON later to an easily-readable format or to a table you can store in a CSV file, for example.

Each platform sets up its APIs in their own way, depending on the types of data or services they provide. This means that each API requires a slightly different strategy as you’re calling and sending queries to it. This requires extra patience and troubleshooting skills.

There are APIs that are very well documented, making them easy to use. Others don’t have good documentation, which means you’ll have to invest time in testing queries and exploring the data on your own.

While the most powerful and flexible way to call APIs and handle output is using code or a platform like Postman, we’ll explore a simpler way using your browser. We’ll be using it to:

call an API with a query, similar to how we visit a website

The first step before starting any API call is to study the documentation: Start a Google search by writing the name of the website you’re scraping and add “+ API documentation”. This information tells you what you can do with the API, which endpoints are available (think of these as different doors that provide separate services or types of data), and what’s the process to make requests.

In this example, we’ll be scraping data from Data USA (documentation here). It’s a free API that doesn’t require authentication, so anyone can call it and see the results immediately. Copy the following example API call to a new tab in your browser and press ENTER to visit it.

https://datausa.io/api/data?drilldowns=Nation&measures=Population

Depending on your browser, the data will be formatted in a different way. Chrome presents the output in raw format, which is the hardest format to read. Firefox formats the data in a more readable format, also letting you interact with the JSON output.

Before moving on, let’s break this request into pieces:

Scores. Stats. Summaries. You can scrape anything online—in a website or an app— then put it on your ScreenCloud signage. Here’s how.

Web Scraping 101: How to Copy Anything for Digital Signage

Web Scraping 101: How to Copy Anything for Digital Signage

Web Scraping 101: How to Copy Anything for Digital Signage

Contents

Ask for an AI Summary