maik
/
Personal-Website
mirror of https://github.com/maikka39/Personal-Website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fix url

Browse Source
pull/4/head
Maik de Kruif 3 years ago
parent 21a20b4b85
commit 7baa940b21
No known key found for this signature in database
GPG Key ID: 46C1200ACD3A432F
1 changed files with 1 additions and 1 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 2
      content/posts/adventofctf/challenge_8.md

2
content/posts/adventofctf/challenge_8.md
Unescape View File

@ -37,7 +37,7 @@ This is talking about robots, which my be a hint to look at the [`robots.txt`](h
### What is a robots.txt file?
A robots.txt file lives at the root of a website. So, for the site <www.example.com>, a robots.txt file would live at <www.example.com/robots.txt>. robots.txt is a plain text file that follows the [Robots Exclusion Standard](http://en.wikipedia.org/wiki/Robots_exclusion_standard#About_the_standard). A robots.txt file consists of one or more rules. Each rule blocks (or allows) access for a given crawler to a specified file path in that website.
A `robots.txt` file lives at the root of a website. So, for the site www.example.com, a robots.txt file would live at www.example.com/robots.txt. robots.txt is a plain text file that follows the [Robots Exclusion Standard](http://en.wikipedia.org/wiki/Robots_exclusion_standard#About_the_standard). A robots.txt file consists of one or more rules. Each rule blocks (or allows) access for a given crawler to a specified file path in that website.
### Opening the file

Write Preview
Loading…
Cancel
Save