Backlinks on SWTORData
Backlinks are now part of SWTORData
Backlinks are when SWTORData automatically links to you, because you link to SWTORData
This system was built to turn SWTORData into a powerful tool other fansites can use to make it easier for players to find information on multiple fansites.
Total links to other fansites on SWTORData: 220121
Unique URL's on other fansites linked to: 38240
New links added within the last hour: 1
New links added within the last 24 hours: 23
The Backlinks system on SWTORData is made up of multiple parts.
As someone that operates a fan sites, it is important you understand how these parts operate and how they will impact your site.
Custom Built Crawler
I looked into several solutions to use other crawlers out there, but none of them could do what I needed. I ended up building my own crawler.
However I can not realistically scan the entire internet for links to myself, so this scanner only looks at approved domains.
Once a day the Crawler will check certain 'key' pages to see if it can find any new pages that need to be looked at. It then adds any new pages to a queue to be processed.
I will manaully tell the Crawler what these key pages are. These should be pages that have most or all links to your content. Best case is a sitemaps.xml file.
Once an hour (At this time) it will pull a certain number of URL's from this queue and go to them. It will then click each link to swtordata, askmrrobot, or torhead.
This triggers what is described in the section below
Note: At this time, it will not rescan a page to find new links.
Impact On Site: Once a day every 'key' page will be requested, once an hour several pages may be requested (I am still figuring out exactly how many to do at once)
The heart of how this entire system works, is recording referral information on a request.
Whenever a user clicks a link on your site, it automatically includes a header with a 'referral' tag that says that the URL on your site is.
This information is recorded for the object that the user clicked. If SWTORData has never seen your URL before, it puts it in a queue to be processed. Otherwise (assuming the domain is approved) the link will appear immediatly on the site.
Every half hour, the queue will be processed. All this does is pull the title from the page so the link can be correctly displayed, and determine if the link should be approved.
Impact On site: Every half an hour, any new URL's found will be requested. Based on my tests, this has generally come out to about 10-15 requests every half an hour per domain.
Joining is very simple.
I track the sites that are currently part of the system. When I see your site appear, I will contact you to get your content added.
Right now you are required to link to an item on SWTORData on every page you want added, however cases where the links are not relevant will not be accepted
You can also email me at email@example.com or contact me on reddit
One of the critical issues of this system, is that it is limited to pages that link to SWTORData.
I am currently in the process of fixing this, but it is taking a bit more time to get out.
I also going to be rolling out an API to make it easier for fan sites to automaticaly link to other relevant pages on other fan sites.
The search will also be improved. Right now the search only looks at the title of the page. However soon it will also look at the content of the page.