What do the sites look like in databases?

Initially, the list that comes to us for verification, has the following form:

  • site1.com
  • site2.com
  • site3.com

Due to the fact that many domains have redirects, which in their turn handle our software, the output list can look like this:

  • http://site1.com
  • http://site2.com/ru/
  • https://site3.com

Thus, the absolute URL path to a specific CMS, including the protocol, is recorded to the database, since one of the frequent redirects is a redirect from http to https. This approach ensures convenient processing of our databases.