A URL isn't being crawled properly / I'm getting a crawl error.

Some sites aggressively monitor for tools like POP and from time to time do a decent job of preventing us from crawling their pages. Fortunately, we have a couple of ways around these types of sites.

The Best Approach

Use the POP Chrome Extension from the POP dashboard to crawl your competitor pages. The POP Chrome Extension can't be blocked. You can find more information on the Chrome Extension here.

A Second Option

You can manually copy and paste your competitor's source code into POP. On the page where you select your competitors you can click the button labeled ‘Show Manual Copy and Paste’. After clicking to copy and paste in the code, a box will appear to insert the source code. There are instructions inside the box in case you aren’t sure what to do. This method is a little on the techy side. Option 1, using the POP Chrome Extension is the recommended method.

I think Cloudflare or a CDN is causing problems

CDNs can cause caching issues from time to time. This can happen with your own site or with a competitor site. The good news is, there are a few ways around the problem.

1.) Use the POP Chrome Extension from the POP dashboard to crawl your page. The POP Chrome Extension gets passed hosting that is redirecting pages. You can find more information on the Chrome Extension here.

2.) Turn off the security feature in your hosting or CDN. Some hosting providers have a dashboard where you can easily turn off this security feature while you are working on your optimization (don’t forget to turn the feature back on once you are finished). Other hosting providers require you to contact support to enable/disable this feature.

Even if the solution in #2 is up your technical alley, we still strongly recommend going with solution #1, The POP Chrome Extension.

3.) You can manually copy and paste your source code into POP. On the page where you select your target URL you will find a checkbox labeled ‘I'd like to copy and paste in html or source code’. Clicking it will reveal a text field to insert the source code. There are instructions inside the box in case you aren’t sure what to do.

4.) The last solution is just time. Usually after 24-48 hours, your hosting will allow POP to run again.

The fixes if your a competitor site is using Cloudflare/CDN and there is an issue (very similar to the fixes above). If your competitor is using Cloudflare/CDN and there is an issue/an error crawling a page, POP will let you know that there was an issue with a particular url so you won't have to guess/hope that things are being crawled correctly. If POP didn't show you an error, POP didn't have any issues crawling a competitor URL.

1.) Use the POP Chrome Extension from the POP dashboard to crawl your competitor pages. The POP Chrome Extension gets passed hosting that is redirecting pages. You can find more information on the Chrome Extension here.

2.) You can manually copy and paste your competitors' source code into POP. This is done by generating a report using the custom method. When prompted, click to choose your own competitors. After choosing them, click the ‘Show Manual Copy and Paste’ button. Each competitor will have a checkbox to paste in their individual source code. Clicking it will reveal a text field to insert the source code. There are instructions inside the box in case you aren’t sure what to do.

3.) The last solution is just time. Usually after 24-72 hours, your competitors' hosting will allow POP to run again.

‍