Data scraping tool for non-coding journalists launches

A tool which helps non-coding journalists scrape data from websites has launched in public beta today.

Import.io lets you extract data from any website into a spreadsheet simply by mousing over a few rows of information.

Until now import.io, which we reported on back in April, has been available in private developer preview and has been Windows only. It is now also available for Mac and is open to all.

Although import.io plans to charge for some services at a later date, there will always be a free option.

The London-based start-up is trying to solve the problem of the fact that there is "lots of data on the web, but it's difficult to get at", Andrew Fogg, founder of import.io, said in a webinar last week.

Those with the know-how can write a scraper or use an API to get at data, Fogg said. "But imagine if you could turn any website into a spreadsheet or API."

Uses for journalists

Journalists can find stories in data. For example, if I wanted to do a story on the type of journalism jobs being advertised and the salaries offered, I could research this by looking at various websites which advertise journalism jobs.

If I were to gather the data from four different jobs boards and enter the information manually into a spreadsheet it would take would take hours if not days; if I were to write a screen scraper for each of the sites it would require knowledge and would probably take a couple of hours. Using import.io I can create a single dataset from multiple sources in a few minutes.

I can then search and sort the dataset and find out different facts, such as how many unpaid internships are advertised, or how many editors are currently being sought.

How it works

When you download the import.io application you see a web browser. This browser allows you to enter a URL for any site you want to scrape data from.

To take the example of the jobs board, this is structured data, with the job role, description and salaries displayed.

The first step is to set up 'connectors' and to do this you need to teach the system where the data is on the page. This is done by hitting a 'record' button on the right of the browser window and mousing over a few examples, in this case advertised jobs. You then click 'train rows'.

It takes between two and five examples to teach import.io where all of the rows are, Fogg explained in the webinar.

The next step is to declare the type of data and add column names. For example there may be columns for 'job title', 'job description' and 'salary'. Data is then extracted into the table below the browser window.

Data from different websites can then be "mixed" into a single searchable database.

In the example used in the webinar, Fogg demonstrated how import.io could take data relating to rucksacks for sale on a shopping website. The tool can learn the "extraction pattern", Fogg explained, and apply that to to another product. So rather than mousing over the different rows of sleeping bags advertised, for example, import.io was automatically able to detect where the price and product details were on the page as it had learnt the structure from how the rucksacks were organised. The really smart bit is that the data from all products can then be automatically scraped and pulled into the spreadsheet. You can then search 'shoes' and find the data has already been pulled into your database.

When a site changes its code a screen scraper would become ineffective. Import.io has a "resilience to change", Fogg said. It runs tests twice a day and users get notified of any changes and can retrain a connector.

It is worth noting that a site that has been scraped will be able to detect that import.io has extracted the data as it will appear in the source site's web logs.

Case studies

A few organisations have already used import.io for data extraction. Fogg outlined three.

    British Red Cross

The British Red Cross wanted to create an iPhone app with data from the NHS Choices website. The NHS wanted the charity to use the data but the health site does not have an API.

By using import.io, data was scraped from the NHS site. The app is now in the iTunes store and users can use it to enter a postcode to find hospital information based on the data from the NHS site.

"It allowed them to build an API for a website where there wasn't one," Fogg said.

    Hewlett Packard

Fogg explained that Hewlett Packard wanted to monitor the prices of its laptops on retailers' websites.

They used import.io to scrape the data from the various sites and were able monitor the prices at which the laptops were being sold in real-time.

    Recruitment site

A US recruitment firm wanted to set up a system so that when any job vacancy appeared on a competitor's website, they could extract the details and push that into their Salesforce software. The initial solution was to write scrapers, Fogg said, but this was costly and in the end they gave up. Instead they used import.io to scrape the sites and collate the data.


Source: http://www.journalism.co.uk/news/data-scraping-tool-for-non-coding-journalists-launches/s2/a554002/

What is data scraping and how can I stop it?

Data scraping (also called web scraping) is the process of extracting information from websites. Data scraping focuses on transforming unstructured website content (usually HTML) into structured data which can be stored in a database or spreadsheet.

The way data is scraped from a website is similar to that used by search bots – human web browsing is simulated by using programs (bots) which extract (scrape) the data from a website.

Unfortunately, there is no efficient way to fully protect your website from data scraping. This is so because data scraping programs (also called data scrapers or web scrapers) obtain the same information as your regular web visitors.

Even if you block the IP address of a data scraper, this will not prevent it from accessing your website. Most data scraping bots use large IP address pools and automatically switch the IP address in case one IP gets blocked. And if you block too many IPs, you will most probably block many of your legitimate visitors.

One of the best ways to protect globally accessible data on a website is through copyright protection. This way you can legally protect the intellectual ownership of your website content.

Another way to protect your site content is to password protect it. This way your website data will be available only to people who can authenticate with the correct username and password.


Source: http://kb.siteground.com/what_is_data_scraping_and_how_can_i_stop_it/

WP Web Scraper

An easy to implement professional web scraper for WordPress. This can be used to display realtime data from any websites directly into your posts, pages or sidebar. Use this to include realtime stock quotes, cricket or soccer scores or any other generic content. The scraper is an extension of WP_HTTP class for scraping and uses phpQuery or xpath for parsing HTML. Features include:

    Can be easily implemented using the button in the post / page editor.
    Configurable caching of scraped data. Cache timeout in minutes can be defined in minutes for every scrap.
    Configurable Useragent for your scraper can be set for every scrap.
    Scrap output can be displayed thru custom template tag, shortcode in page, post and sidebar (through a text widget).
    Other configurable settings like timeout, disabling shortcode etc.
    Error handling - Silent fail, error display, custom error message or display expired cache.
    Clear or replace a regex pattern from the scrap before output.
    Option to pass post arguments to a URL to be scraped.
    Dynamic conversion of scrap to specified character encoding (using incov) to scrap data from a site using different charset.
    Create scrap pages on the fly using dynamic generation of URLs to scrap or post arguments based on your page's get or post arguments.
    Callback function to parse the scraped data.

For demos and support, visit the WP Web Scraper project page. Comments appreciated.

Tags: curl, html, import, page, phpquery, Post, Realtime, sidebar, stock market, web scraping, xpath   



Source: http://wordpress.org/plugins/wp-web-scrapper/

Simple method of Data Scrapping

There are so many tools available on the Internet are scraping data. With these tools, without stress, you can download a large amount of data. The last decade, the Internet revolution as an information center was the world. You can get any information on the Internet. However, if you want to work with specific information, you must find other sites. Download all the information on the website that interests you, then you must copy the information in the document header. Everything seems to work a bit "more difficult. With scraping tools, your time, save money and can reduce manual labor.

Tools for extracting Web data to extract data from HTML pages and Web sites to compare data. Each day, there are many sites are hosted on the Internet. You can not see all the sites the same day. These data mining tools, you can view all pages on the Internet. If you use a wide range of applications, the scraping tool is also useful for you.

Software tools for data retrieval for structured data that is used on the Internet. There are so many Internet search engines to help you find a site for a particular problem would be. Various sites, the data appears in different styles. The expert scraped help you compare the different sites and structures for recording updated data.

And the web crawler software tool is used to index the Web pages on the Internet, moving data to the Internet from your hard drive. With this work, you can surf the Internet much faster than they are connected. It is time to use the tip of the device is important if you try to download data from the Internet. It will take considerable time to download. However, the device with faster Internet rate. There you can download all the corporate data of the person is another tool called e-mail extractor. The tribute, you can easily target your e-mail client. Each time your product is able to send targeted advertisements to customers. The customer database to find the best equipment.

Scraping and data extraction can be used in any organization, corporation, or any company which is a data set targeted customer industry, company, or anything that is available on the net as some data, such as e-ID mail data, site name, search term or what is available on the web. In most cases, data scraping and data mining services, not a product of industry, are marketed and used for example to reach targeted customers as a marketing company, if company X, the city has a restaurant in California, the software relationship that the city's restaurants in California and use that information for marketing your product to market-type restaurant company can extract the data.

MLM and marketing network using data mining and data services to each potential customer for a new client by extracting the data, and call customer service, postcard, e-mail marketing, and thus produce large networks to send large groups of construction companies and their products.

However, there are tolls are scraping on the Internet. And some sites have reliable information about these tools. By paying a nominal amount to download these tools.


Source: http://goarticles.com/article/Simple-method-of-Data-Scrapping/4692026/

Google scraper to download data from Google search pages

Web scraping involves extraction of data from websites and converting them to usable format. There are many web scraping tools designed specific purposes like white pages scraper, amazon scraper, email address scraper, customer contract scraper etc. Google scraper is one such web scraping application which is used to extract google search results. This application will gather useful information from search results of Google which can be helpful in preparation of prospective databases with potential customers, email lists, online price comparison, real estate data, job posting information and customer demographics. Many people nowadays use web scraping to minimize the effort involved in manual extraction of data from websites.

You can find the details of customers in particular locality be searching through the white pages of that region. Also, if you want to gather email address or phone numbers of customers, you can do that with email address extractor. Google scraper will be useful to scrape google results and store them in text file, Spread sheets or database. The data scraping is automated function done by software application to extract data from websites by simulation human exploration of web through scripts like Perl, Python, and JavaScript etc. The data scraping could be great tool for programmers and can have lot of value for the money.

Also data collected through web scraping tool is accurate and ensures faster results. You can use this to collect email address of potential customers for your email marketing campaign to promote your products. You can search for relevant information about customer products. If you want to download images of products you can just enter the relevant keyword and google scraper will automatically extract the data from you google images page. You can generate sales leads and expand your business by using scraping tools which can save lot of time and money.



Source: http://goarticles.com/article/Google-scraper-to-download-data-from-Google-search-pages/4254108/

Simple Answer to a Frequently Asked Question, ‘What Is Screen Scraping’?

Undoubtedly, data extraction today has become a laborious task and thus calls the demand for latest technology to accomplish the job. With the support of web screen scraping services, the job to drag out required data and information has become simple and easy. Now the questions arises âEUR~what is screen scrapingâEUR(TM)? Well, it is a specially designed program that has proved to be of great help for the purpose of extraction of data, images and heavy files as well. This software helps individuals to download the specific data in the desired format. This service is like a boon for many websites.

There lies a tough competition in the market today. Business entrepreneurs are trying hard to get beneficial outcome in their business growth. With the support and help of scraping services, business owners are extracting the information of many internet users in their website and this readily helps them to grow their business. One big advantage of this program is that it can develop tons of datas in less time. In business scenario, it is time that matters a lot. So, businesses today are making use of this service to get the data available in no time.

Benefits of Screen Scraping

Fast Scraping: One greatest advantage of using this software is that it saves your time and labor. It lessens the chances of making you wait for long hours to provide you data. Also, the quick scraping tools offer you latest data.

Presentable: Scraping programs also offers data in readable format which could be used in a hassle free manner. The service providers can provide data in database file or spreadsheet or any other format as desired by the user. Data which cannot be read is of no use. Presentation means a lot.

As screen scarping is a software, it is made. In its development involves a group of experts that possess great knowledge in the field. They are basically programmers who have gained great expertise in the domain and are efficient to load innumerable dataâEUR(TM)s from different websites in very little time.

Today, the market is swarming up with various service providers offering screen scraping services. Explore different websites and select one that excites you the most. Going online would not only save your time but also reduce the difficulty of going out in the sweltering sun. Get the details of the firm and contact their service providers to get the data extracted for your business. Furthermore, if you are concerned about the charges, do not worry as the facilities can be availed at realistic rates.

Henceforth, give your business a new turn with the best screen scraping service providers.



Source: http://goarticles.com/article/Simple-Answer-to-a-Frequently-Asked-Question-What-Is-Screen-Scraping/7872372/

Screen Scraper Software

Applications for Monitoring Competitor Pricing by using screen scraping.

In a world with seamless integration of internet information, more and more web data extraction services can be found providing reliable ways to monitor competitive pricing for your business. In addition to streamlining content, these companies gather resourceful information. Which is of course a vital asset for any company or private group's use. Not only for collecting and refining web content, you can also make use of gathered information in an organized form for purposes of intelligence, study, and storage for future use. Finding this form of web extraction service for you can take some seriously contemplated decision making, if you don't know where to look. But, with this article you will hopefully find that deciding which one best suites your need doesn't have to be headache inducing in the end.

The first name that comes to mind for monitoring competitor pricing would have to be Mozenda. Being the highest rated on sites like theeasybee.com, they have become a optimal solution for web content scraping of this nature. Mozenda offers a extremely easy, and organized approach with it's carefully crafted user interface. Collecting detailed marketing and research data could not be made simpler than they have made it. Dedicated to the search of online content for projects like competitive pricing, lead generation, or scientific research, you will find that Mozenda has been designed to fit all of your web extraction needs. But this is only a mere glimpse of what it has to offer. Mozenda converts your collected web data into many useful formats like CSV, TSV, XML, and RSS just to name a few. Also, for those new to web extraction, they even offer to set up your first project free of charge. But, I doubt you would even need that with all of the resources made available to you. They have a section on their page offering instructional videos that show you how to set up your very own projects extremely quick, and easily. In addition to the already impressive capabilities of Mozenda's software, they offer many sub services in order to get your job done correctly as well. Giving you more time to actually use the information collected in your projects any manner you like.

At a not too distant second is Kapow Technologies. Proudly claiming to deliver business solutions involving web data in only a fraction of the time as their competitors in software development. They also boast the ability to achieve the same end results in only a fraction of cost as well. Having gained much acclaim with their partnership with IBM in order to create a Web 2.0 Expo application for the IPhone in less than three hours, they definitely have the expertise to carry out the much simpler project ideas like these. One major attraction to their applications are it's abilities to extract with absolutely no coding, through it's exclusive point-and-click develop technology. They are a unique enterprise, capable of wrapping any existing web content or API with this lossless technique.

To see which applications and services work best for you, it is highly suggested that you take advantage of the free trial downloads that are made available on these sites. Most come with a two week test period, which allows more than enough time to figure out which one is best suited for your optimal business performance. Monitoring your competitor's pricing has been made a extremely easy task with all of the accessible options. Luckily, tedious and time-consuming methods are completely a thing of the past.



Source: http://goarticles.com/article/Screen-Scraper-Software/3623340/