HTTPS, Google and indexing

served to prepare the ground for the big announcement that arrived on 8 September 2016 directly on the Google Security Blog: with the release of Chrome 56 (1 January 2017) HTTP sites will be marked by the browser itself as "not secure", information that will be clearly visible to users who browse these sites and which will constitute a strong penalty for the sites themselves.

Let's start with the basic definitions and then continue with the analysis of the various steps that, over the last 6 years, have led Google to make this important decision.

What is the HTTPS protocol? We continue by directly quoting the primary source, namely Google and its guide dedicated to webmasters:

"HTTPS (Hypertext Transfer Protocol Secure) is an Internet communication protocol that protects the integrity and confidentiality of user data exchanged between computers and sites. For example, when a user enters data into a form on your site to sign up for updates or to purchase a product, HTTPS protects the user's personal information that the user provides to the site. Users expect to be able to provide data via a website securely […].

Data sent via HTTPS is protected using the Transport Layer Security (TLS) protocol, which provides three basic layers of protection:

Cryptography. The data exchanged is encrypted to protect it from interception. This means that, while the user consults a website, no one can "listen" to his conversations, track his activities on multiple pages or steal his information

Data integrity. Data cannot be modified or damaged in transit, intentionally or unintentionally, without detection

Authentication . It proves that users are communicating with the intended website. It protects against man-in-the-middle attacks and instills trust in users, which translates into other business benefits."

Now let's make a brief summary of all the most significant steps that led to the decision that will be implemented starting next year.

For Google the issue of security has always been absolutely fundamental. In fact, since 2010 they have introduced the HTTPS protocol by default within Gmail and in many of their other services (SSL Search, Docs and Google+). Mainly this means that all people who use Google Search, Gmail email and Google Drive sharing, for example, automatically have access to a secure connection directly with Google.

But Big G didn't stop there. In August 2014, with a post on the blog dedicated to Webmasters, they declared that their intent is to work so that the vast world of the Internet becomes a safer place on a general level. A big part of this goal is to make sure that websites accessed through Google are safe. To achieve this objective, a real guide dedicated to webmasters was created which, among the many topics covered, also explains how to fix and avoid security errors on their sites.

HTTPS Everywhere In the same year, during the Google I/O conference (the annual conference held in San Francisco aimed at web developers and focused on the development of web and mobile applications), Ilya Grigorik (web performance engineer) and Pierre Far (Senior webmaster trends analyst) held a talk dedicated entirely to web security, entitled " HTTPS everywhere ". In summary, during this speech the two experts tried to convince those present that all communications should take place in a secure manner by default. During the speech, the importance of protecting the security, privacy and integrity of user data was highlighted several times. A real guided lesson on how to make your websites secure by default, indicating which technologies are necessary for this purpose, the configurations and best practices to follow on how to migrate a site to HTTPS and make it "user and search friendly".

https://youtu.be/cBhZ6S0PFCY

Guide to using HTTPS The first tests to include the use of the HTTPS protocol as an indexing factor in the search results algorithm began in 2014. At the time, this factor influenced less than 1% of global queries and certainly had less weight than other factors, first and foremost high-quality content. To encourage the use of the HTTPS protocol among webmasters, here is the guide to the "Best Practices for using HTTPS" directly available in the Google Help Center. The guide contains all the information necessary to optimize the use of the HTTPS protocol, but above all the instructions for migrating your site from HTTP to HTTPS. Among the list of best practices, we have decided to mention a few:

Request an SSL certificate for the website by choosing between dedicated, multi-domain or wildcard certificates (we recommend using certificates with a 2048-bit key) and install the chosen SSL certificate on the web server.

Redirect users and search engines to the HTTPS page or resource using a server-side 301 redirect. [see also the guide "Moving a site with URL changes"]

Do not block the site in HTTPS using the robots.txt file

Obviously these are just some of the points to follow to better optimize your HTTPS website and therefore improve its indexing and reputation.

SSL Certificate Let's focus for a moment on the SSL Certificate issue, which concerns us at Colt Engine (Joomlahost and Dnshosting) very closely. We have seen that, in order to encrypt browsing and make it safe for users, a website must install a valid certificate which is then presented to the browser during browsing and which confirms its status as a safe site. The certificates necessary for this type of operation are issued by a trusted certification authority (CA). Google has decided to launch a specific project dedicated to certificates and certification authorities, the Certificate Transparency project, "designed to protect the certificate issuing process by offering an open framework for monitoring and controlling HTTPS certificates". On the project's official website, its main purpose is indicated: "the project fixes several structural flaws in the SSL certificate system […] These flaws weaken the reliability and effectiveness of connections and can compromise critical TLS/SSL mechanisms, including domain validation, end-to-end encryption and chains of trust established by certificate authorities. If left unchecked, these flaws can facilitate a wide range of security attacks […]" . The Certificate Transparency project helps identify invalid SSL certificates and, more importantly, detect fraudulent certificate authorities to protect users from insecure websites. It is therefore clear that Google aims to protect users while browsing the web, at the cost of penalizing websites that do not respect certain characteristics. In the section dedicated to the Certificate Transparency project present in the Transparency Report, Google invites all CAs to write the certificates issued in a specific tamper-proof log. “In the future, Chrome and other browsers may decide not to accept certificates that have not been written to these types of logs.”

The importance of having an SSL certificate installed correctly on your website therefore allows the site itself to give a guarantee of security to its users, a fact that Google greatly appreciates by repaying the site with good indexing and a series of features that we will see in detail later.

HTTPS as a ranking factor We continue our journey through the gradual implementation of the HTTPS protocol as a relevant factor in the ranking of a website in the Google algorithm. In 2015, Google's indexing system was "recalibrated" to search for as many HTTPS pages as possible. Specifically, Google's crawl began to search for all those apparently identical URLs but actually served by the two different protocols and it was decided to favor indexing of pages in HTTPS if these:

they did not contain any unsafe dependencies

they were not blocked by the robots.txt file

they did not redirect users to an HTTP page

they did not contain a rel="canonical" link to the page in HTTP

they did not contain the noindex meta tag

In addition to these characteristics, Google's crawl began to evaluate the presence of a valid and correctly installed SSL certificate on the server and the presence of URLs with HTTPS in the site's sitemap.

Through this review of the algorithm, Google has begun to index pages managed in HTTPS by default, favoring their positioning over their HTTP versions.

Today, over 50% of web pages are in HTTPS. So we come to what is happening today, specifically we are referring to an article that appeared on November 3rd on the Google Security Blog. In summary, in this blog written by the Chrome Security Team, reference is made to the fact that "a web with ubiquitous HTTPS is not such a distant hypothesis. It is already happening now, especially for those who use Chrome which offers secure browsing as standard". On the same day this article was published, a new section entirely dedicated to the HTTPS protocol was added to Google's Transparency Report (a set of data that clarifies the influence of laws and regulations on Internet users and the flow of information online).

“More than half of Chrome desktop users' total page loads and two-thirds of desktop users' total time are through HTTPS, and we expect these numbers to continue their rapid growth trajectory.”

Within the Transparency Report, Google's desire to "completely encrypt its products and services" is explicitly stated.

"We are trying to implement HTTPS on all our products. In March 2014 we announced the exclusive use of HTTPS for Gmail. We are working to overcome the technical barriers that complicate encryption support on some of our products", this is the statement taken directly from the document complete with a supporting graph, which highlights the traffic broken down by specific product:

To encourage site migrations from HTTP to HTTPS, new features have been added to the use of this protocol, beyond greater security. In addition to offering the best web performance, HTTPS offers powerful features that benefit conversions on a site, including: Service Workers for offline support and Web Push Notifications. In addition to existing features such as credit card autofill and geolocation APIs in HTML5, features that are too powerful to be used with insecure HTTP browsing.

To help users navigate the web safely, Chrome itself has always indicated secure connections through the use of an icon in the address bar. To date, Google's browser has never explicitly indicated HTTP connections as insecure, but as we said at the beginning of our article, starting from January 2017 with Chrome 56, this will no longer be true: all HTTP pages that contain password entry fields or credit card information will be clearly marked as insecure. Google itself has stated that this first operation will be only part of a long-term plan whose final aim is to mark all HTTP sites as unsafe.

Another aspect of this long-term plan lies in the study of the new icons that will be used by the new Chrome 56 to indicate the lack of security when using an HTTP connection on a website. In fact, recent studies show that users do not perceive the non-presence of a security icon as an alarm and, furthermore, they also become accustomed to alerts and warnings that appear too frequently. These considerations are the subject of the paper "Rethinking Connection Security Indicators" presented on the occasion of the Twelfth Symposium on Usable Privacy and Security (22-24 June 2016 – Denver USA). Within this research, a new set of browser security indicators was proposed, based on user research and an understanding of the design challenges faced by different browsers. To motivate the need to create new security indicators, all the weak points of the current icons were highlighted on the basis of an interview with a sample of 1329 people on the current indicators used by Google Chrome. 40 icons were identified and an interview was carried out to evaluate their perception. Ultimately, 3 specific indicators were selected to signal security and lack of security on a website: "our proposed indicators have been adopted by Google Chrome, and we hope to motivate other browsers to update theirs too", thus concludes the abstract of the paper which underlines how user perception is an absolutely fundamental factor in pushing webmasters to migrate their websites to the HTTPS protocol in order to offer a better service to the users themselves.

The data on the increase in traffic to HTTPS pages, the study of the perception of more significant icons to signal security (or the lack of it) on a website, the implementation on all Google products and the addition of useful features for commercial purposes, make it clear the extent of the investment towards transforming the web into a safer place for users. Just as, equally clearly, the intention to penalize all sites that continue to use the non-secure HTTP protocol is evident.

Having elaborated these considerations, we conclude this overview of the HTTPS security protocol with a necessary mention of the hosting service that hosts your website. Among the many opportunities that are currently on the market, after reading this article, you will have understood the importance of choosing a Hosting Service Provider that gives you the possibility of adding, already when purchasing your service, a valid SSL Certificate that allows you to create the HTTPS version of your website. As in many cases similar to this, the rule of "prevention is better than cure" always applies and acting in time can give your website an extra boost to climb rankings in search results, especially Google.

Have you already migrated your website to HTTPS?