Recently, I had to troubleshoot several websites where I had to make HTTP GET/POST requests from a remote server. The problem was that security plugins were blocking a lot of them.
As a site owner, I understand the need for these plugins. In fact, they are essential to have these days with WordPress based sites being attacked everyday. As a developer, however, they can make life hard.
In this post, I am going to discuss the lessons and methods I learned while dealing with the security plugins.
Document Everything, In Detail
If you are in a hurry or don’t plan to read ahead (you should!), just take this point away and you’ll be good! Creating a well documented support page for your users is going to help a lot.
For bonus points, mention the support page or better, the troubleshooting steps in the error message to make sure that users see it.
I did this recently when an AppSumo promotion caused a flood of curation users for EpicBeat. Now this won’t win the support page of the year award, but it got the job done and several customers ended up solving the issue by themselves, reducing the support work.
Give Details in Error Messages
It will greatly help if your error message can tell the user exactly what went wrong instead of giving a cryptic message (I’m guilty of doing this a lot).
Note down the common errors that you see and create error messages for them. This will save a lot of time going forward.
For reference, here are some common behaviours with security plugins:
- WP Spamshield: It will return a 403 error saying the request was blocked and will mention the plugin name.
- iThemes Security: It will also return a 403 error but won’t mention the plugin name. Here’s how the response looks like:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access /v1/validate\non this server.<br />\n<br />\nPossible causes of this error include:</p>\n<ol>\n<li>The request was forbidden by rules in the .htaccess file.</li>\n<li>The directory you requested does not have an index.html or index.php file.</li>\n<li>The permissions on the file or directory are incorrect.</li>\n</ol>\n<p>For details about why the request was forbidden, see the Apache error log at:</p>\n<pre>/srv/users/SYSUSER/log/APPNAME/APPNAME_apache.error.log</pre>\n<p><br /><br /><b>\n</b></p>\n</body></html>\n |
Sometimes, you may see 5xx range from other plugins. Remember, if you see a cryptic error with a plain page, it’s almost always a security plugin.
Look At How You Send Requests
Last week, I ended up troubleshooting one website where iThemes security was blocking the requests. Even disabling the plugin didn’t help.
After spending some time, I found that the plugin was adding rules to .htaccess and they remained even after plugin got disabled.
After diving into .htaccess file, I found the exact line responsible for the issue:
RewriteRule ^.* - [F,L]
This line denies access with a 403 (that’s what F flag does) for all requests. Finally, I discovered that this was happening because we had set no User-Agent while sending request. 1
To fix this, you can set a user agent while making the request in curl. Here’s a quick example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <?php | |
| $ch = curl_init(); | |
| curl_setopt($ch,CURLOPT_URL, $url); | |
| curl_setopt($ch,CURLOPT_POST, count($this->data)); | |
| curl_setopt($ch, CURLOPT_USERAGENT, 'Name of your service'); // this is where user agent is set |
You can also use wp_safe_remote_post and other functions which take care of a lot of similar issues for you. 2
Anything else?
What issues have you encountered in similar positions? Do comment and let me know of other plugin behaviours.
- Refer to these guides if you want to know more about Rewrite Rules: URL Rewriting for Beginners, Apache Rewriting Guide. ?
- Docs for wp_safe_remote_post ?
Hi and thanks for this post. I have iThemes Security installed and it has severely affected my SEO with Google Webmaster tools. 40+ of my pages are not indexed and are blocked with a 403 error.
I tried removing the RewriteRule ^.* – [F,L] and it caused an internal 500 error so I had to replace it in the htaccess file.
I am not a developer or programmer like you, so I do not understand anything about “curl.”
It appears you have a solution to the 403 iThemes Security issue, but I simply have no clue as to what you are talking about.
Also, how do I solve the problem with Google Webmaster Tools and Yoast SEO Search Console? They both mirror the exact same issue and I have no clue how to unblock my pages so Google can index them?
Any advice would be helpful.
Thanks!