Compare commits

..

1 Commits

Author SHA1 Message Date
Alicia Sykes
bc45b91b51 Refactored Dockerfile 2023-08-29 21:47:39 +01:00
51 changed files with 2885 additions and 3431 deletions

29
.env
View File

@@ -2,24 +2,11 @@
# Be sure to uncomment any line you populate # Be sure to uncomment any line you populate
# Everything is optional, but some features won't work without external API access # Everything is optional, but some features won't work without external API access
# API Keys for external services (backend) # GOOGLE_CLOUD_API_KEY=''
GOOGLE_CLOUD_API_KEY='' # SHODAN_API_KEY=''
TORRENT_IP_API_KEY='' # REACT_APP_SHODAN_API_KEY=''
SECURITY_TRAILS_API_KEY='' # WHO_API_KEY=''
BUILT_WITH_API_KEY='' # REACT_APP_WHO_API_KEY=''
URL_SCAN_API_KEY='' # SECURITY_TRAILS_API_KEY=''
TRANCO_USERNAME='' # BUILT_WITH_API_KEY=''
TRANCO_API_KEY='' # CI=false
CLOUDMERSIVE_API_KEY=''
# API Keys for external services (frontend)
REACT_APP_SHODAN_API_KEY=''
REACT_APP_WHO_API_KEY=''
# Configuration settings
# CHROME_PATH='/usr/bin/chromium' # The path the the Chromium executable
# PORT='3000' # Port to serve the API, when running server.js
# DISABLE_GUI='false' # Disable the GUI, and only serve the API
# API_TIMEOUT_LIMIT='10000' # The timeout limit for API requests, in milliseconds
# API_CORS_ORIGIN='*' # Enable CORS, by setting your allowed hostname(s) here
# REACT_APP_API_ENDPOINT='/api' # The endpoint for the API (can be local or remote)

217
.github/README.md vendored
View File

@@ -19,13 +19,11 @@
- [Mirror](#mirror) - [Mirror](#mirror)
- [Features](#features) - [Features](#features)
- **[Usage](#usage)** - **[Usage](#usage)**
- [Deployment](#deployment)
- [Option#1: Netlify](#deploying---option-1-netlify)
- [Option#2: Vercel](#deploying---option-2-vercel)
- [Option#3: Docker](#deploying---option-3-docker)
- [Option#4: Source](#deploying---option-4-from-source)
- [Configuration Options](#configuring)
- [Developer Setup](#developing) - [Developer Setup](#developing)
- [Deploying, Option#1: Netlify](#deploying---option-1-netlify)
- [Deploying, Option#2: Docker](#deploying---option-2-docker)
- [Deploying, Option#3: Source](#deploying---option-3-from-source)
- [Configuration Options](#configuring)
- **[Community](#community)** - **[Community](#community)**
- [Contributing](#contributing) - [Contributing](#contributing)
- [Bugs](#reporting-bugs) - [Bugs](#reporting-bugs)
@@ -58,17 +56,10 @@ A hosted version can be accessed at: **[web-check.as93.net](https://web-check.as
### Mirror ### Mirror
The source for this repo is mirrored to CodeBerg, available at: **[codeberg.org/alicia/web-check](https://codeberg.org/alicia/web-check)** The source for this repo is mirrored to CodeBerg, available at: **[codeberg.org/alicia/web-check](https://codeberg.org/alicia/web-check)**
### Status ### Motivation
Often when you're looking into a website, there's several things you always initially check.
Build & Deploys: [![Netlify Status](https://api.netlify.com/api/v1/badges/c43453c1-5333-4df7-889b-c1d2b52183c0/deploy-status)](https://app.netlify.com/sites/web-check/deploys)
[![Vercel Status](https://therealsujitk-vercel-badge.vercel.app/?app=web-check-ten)](https://vercel.com/as93/web-check/)
[![🐳 Build + Publish Docker Image](https://github.com/Lissy93/web-check/actions/workflows/docker.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/docker.yml)
[![🚀 Deploy to AWS](https://github.com/Lissy93/web-check/actions/workflows/deploy-aws.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/deploy-aws.yml)
<br />
Repo Managament & Miscellaneous: [![🪞 Mirror to Codeberg](https://github.com/Lissy93/web-check/actions/workflows/mirror.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/mirror.yml)
[![💓 Inserts Contributors & Sponsors](https://github.com/Lissy93/web-check/actions/workflows/credits.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/credits.yml)
None of this is hard to find with a series of basic curl commands and NPMAP plus a combination of online tools. But it's just so much easier to have everything done all at once, presented clearly and visible in one place :)
### Features ### Features
@@ -739,102 +730,14 @@ This may be useful to see what a given website looks like, free of the constrain
</details> </details>
Read more here: **[web-check.xyz/about](https://web-check.xyz/about)** Read more here: **[web-check.as93.net/about](https://web-check.as93.net/about)**
_Note that not all checks will work for all sites. Sometimes it's not possible to determine some information, and the demo instance has some limitations imposed by Netlify for the lambda functions._
--- ---
## Usage ## Usage
### Deployment
### Deploying - Option #1: Netlify
Click the button below, to deploy to Netlify 👇
[![Deploy to Netlify](https://img.shields.io/badge/Deploy-Netlify-%2330c8c9?style=for-the-badge&logo=netlify&labelColor=1e0e41 'Deploy Web-Check to Netlify, via 1-Click Script')](https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check)
### Deploying - Option #2: Vercel
Click the button below, to deploy to Vercel 👇
[![Deploy with Vercel](https://img.shields.io/badge/Deploy-Vercel-%23ffffff?style=for-the-badge&logo=vercel&labelColor=1e0e41)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flissy93%2Fweb-check&project-name=web-check&repository-name=web-check-fork&demo-title=Web-Check%20Demo&demo-description=Check%20out%20web-check.xyz%20to%20see%20a%20live%20demo%20of%20this%20application%20running.&demo-url=https%3A%2F%2Fweb-check.xyz&demo-image=https%3A%2F%2Fraw.githubusercontent.com%2FLissy93%2Fweb-check%2Fmaster%2F.github%2Fscreenshots%2Fweb-check-screenshot10.png)
### Deploying - Option #3: Docker
Run `docker run -p 3000:3000 lissy93/web-check`, then open [`localhost:3000`](http://localhost:3000)
<details>
<summary>Docker Options</summary>
You can get the Docker image from:
- DockerHub: [`lissy93/web-check`](https://hub.docker.com/r/lissy93/web-check)
- GHCR: [`ghcr.io/lissy93/web-check`](https://github.com/Lissy93/web-check/pkgs/container/web-check)
- Or build the image yourself by cloning the repo and running `docker build -t web-check .`
</details>
### Deploying - Option #4: From Source
Install the prerequisites listed in the [Developing](#developing) section, then run:
```bash
git clone https://github.com/Lissy93/web-check.git # Grab the code
cd web-check # Move into the project directory
yarn install # Install dependencies
yarn build # Build the app for production
yarn serve # Start the app (API and GUI)
```
---
### Configuring
By default, no configuration is needed.
But there are some optional environmental variables that you can set to give you access to some additional checks, or to increase rate-limits for some checks that use external APIs.
**API Keys & Credentials**:
Key | Value
---|---
`GOOGLE_CLOUD_API_KEY` | A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
`REACT_APP_SHODAN_API_KEY` | A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
`REACT_APP_WHO_API_KEY` | A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
<details>
<summary><small>Full / Upcoming Vals</small></summary>
- `GOOGLE_CLOUD_API_KEY` - A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
- `REACT_APP_SHODAN_API_KEY` - A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
- `REACT_APP_WHO_API_KEY` - A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
- `SECURITY_TRAILS_API_KEY` - A Security Trails API key ([get here](https://securitytrails.com/corp/api)). This will show org info associated with the IP
- `CLOUDMERSIVE_API_KEY` - API key for Cloudmersive ([get here](https://account.cloudmersive.com/)). This will show known threats associated with the IP
- `TRANCO_USERNAME` - A Tranco email ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `TRANCO_API_KEY` - A Tranco API key ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `URL_SCAN_API_KEY` - A URLScan API key ([get here](https://urlscan.io/)). This will fetch miscalanious info about a site
- `BUILT_WITH_API_KEY` - A BuiltWith API key ([get here](https://api.builtwith.com/)). This will show the main features of a site
- `TORRENT_IP_API_KEY` - A torrent API key ([get here](https://iknowwhatyoudownload.com/en/api/)). This will show torrents downloaded by an IP
</details>
**Configuration Settings**:
Key | Value
---|---
`CHROME_PATH` | The path the the Chromium executable (e.g. `/usr/bin/chromium`)
`PORT` | Port to serve the API, when running server.js (e.g. `3000`)
`DISABLE_GUI` | Disable the GUI, and only serve the API (e.g. `false`)
`API_TIMEOUT_LIMIT` | The timeout limit for API requests, in milliseconds (e.g. `10000`)
`REACT_APP_API_ENDPOINT` | The endpoint for the API, either local or remote (e.g. `/api`)
All values are optional.
You can add these as environmental variables. Either put them directly into an `.env` file in the projects root, or via the Netlify / Vercel UI, or by passing to the Docker container with the --env flag, or using your own environmental variable management system
Note that keys that are prefixed with `REACT_APP_` are used client-side, and as such they must be scoped correctly with minimum privileges, since may be made visible when intercepting browser <-> server network requests
---
### Developing ### Developing
1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git` 1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git`
@@ -845,6 +748,64 @@ Note that keys that are prefixed with `REACT_APP_` are used client-side, and as
You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/). You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/).
Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present. Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present.
### Deploying - Option #1: Netlify
Click the button below, to deploy to Netlify 👇
[![Deploy to Netlify](https://img.shields.io/badge/Deploy-Netlify-%2330c8c9?style=for-the-badge&logo=netlify&labelColor=1e0e41 'Deploy Web-Check to Netlify, via 1-Click Script')](https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check)
### Deploying - Option #2: Docker
Run `docker run -p 8888:3000 lissy93/web-check`, then open `http://localhost:3000`
You can get the Docker image from:
- DockerHub: [`lissy93/web-check`](https://hub.docker.com/r/lissy93/web-check)
- GHCR: [`ghcr.io/lissy93/web-check`](https://github.com/Lissy93/web-check/pkgs/container/web-check)
- Or build the image yourself by cloning the repo and running `docker build -t web-check .`
### Deploying - Option #3: From Source
Follow the instructions in the [Developing](#developing) section above, then run `yarn build` && `yarn serve` to build and serve the application.
The following commands will work:
```bash
git clone https://github.com/Lissy93/web-check.git # Grab the code
cd web-check # Move into the project directory
yarn install # Install dependencies
yarn build # Build the app for production
yarn serve # Start the app (API and GUI)
```
### Configuring
By default, no configuration is needed.
But there are some optional environmental variables that you can set to give you access to some additional checks, or to increase rate-limits for some checks that use external APIs.
Note that keys that are prefixed with `REACT_APP_` are used client-side, and as such they must be scoped correctly with minimum privileges.
**API Keys & Credentials**:
- `GOOGLE_CLOUD_API_KEY` - A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
- `REACT_APP_SHODAN_API_KEY` - A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
- `REACT_APP_WHO_API_KEY` - A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
- `SECURITY_TRAILS_API_KEY` - A Security Trails API key ([get here](https://securitytrails.com/corp/api)). This will show org info associated with the IP
- `CLOUDMERSIVE_API_KEY` - API key for Cloudmersive ([get here](https://account.cloudmersive.com/)). This will show known threats associated with the IP
- `TRANCO_USERNAME` - A Tranco email ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `TRANCO_API_KEY` - A Tranco API key ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `URL_SCAN_API_KEY` - A URLScan API key ([get here](https://urlscan.io/)). This will fetch miscalanious info about a site
- `BUILT_WITH_API_KEY` - A BuiltWith API key ([get here](https://api.builtwith.com/)). This will show the main features of a site
- `TORRENT_IP_API_KEY` - A torrent API key ([get here](https://iknowwhatyoudownload.com/en/api/)). This will show torrents downloaded by an IP
**Configuration Settings**:
- `CHROME_PATH` (e.g. `/usr/bin/chromium`) - The path the the Chromium executable
- `PORT` (e.g. `3000`) - Port to serve the API, when running server.js
- `DISABLE_GUI` (e.g. `false`) - Disable the GUI, and only serve the API
- `API_TIMEOUT_LIMIT` (e.g. `10000`) - The timeout limit for API requests, in milliseconds
- `REACT_APP_API_ENDPOINT` (e.g. `/api`) - The endpoint for the API (can be local or remote)
The above can be added into an `.env` file in the projects root, or via the Netlify UI, or by passing directly to the Docker container with the --env flag.
All variables are optional.
--- ---
@@ -898,17 +859,10 @@ Credit to the following users for contributing to Web-Check
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/JinnaBalu"> <a href="https://github.com/gso-trifork-security">
<img src="https://avatars.githubusercontent.com/u/11784253?v=4" width="80;" alt="JinnaBalu"/> <img src="https://avatars.githubusercontent.com/u/69247026?v=4" width="80;" alt="gso-trifork-security"/>
<br /> <br />
<sub><b>Jinna Balu</b></sub> <sub><b>Gustav Soelberg</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/liss-bot">
<img src="https://avatars.githubusercontent.com/u/87835202?v=4" width="80;" alt="liss-bot"/>
<br />
<sub><b>Alicia Bot</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
@@ -924,8 +878,7 @@ Credit to the following users for contributing to Web-Check
<br /> <br />
<sub><b>Ulises Gascón</b></sub> <sub><b>Ulises Gascón</b></sub>
</a> </a>
</td></tr> </td>
<tr>
<td align="center"> <td align="center">
<a href="https://github.com/abhishekMuge"> <a href="https://github.com/abhishekMuge">
<img src="https://avatars.githubusercontent.com/u/49590582?v=4" width="80;" alt="abhishekMuge"/> <img src="https://avatars.githubusercontent.com/u/49590582?v=4" width="80;" alt="abhishekMuge"/>
@@ -945,14 +898,14 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<tr> <tr>
<td align="center"> <td align="center">
<a href="https://github.com/emlazzarin"> <a href="https://github.com/emlazzarin">
<img src="https://avatars.githubusercontent.com/u/1141361?u=714e3487a3f2e0df721b01a0133945f075d3ff68&v=4" width="80;" alt="emlazzarin"/> <img src="https://avatars.githubusercontent.com/u/1141361?v=4" width="80;" alt="emlazzarin"/>
<br /> <br />
<sub><b>Eddy Lazzarin</b></sub> <sub><b>Eddy Lazzarin</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/AnandChowdhary"> <a href="https://github.com/AnandChowdhary">
<img src="https://avatars.githubusercontent.com/u/2841780?u=ca8e292b15abcc6cddaeae0abded0115c51b4789&v=4" width="80;" alt="AnandChowdhary"/> <img src="https://avatars.githubusercontent.com/u/2841780?v=4" width="80;" alt="AnandChowdhary"/>
<br /> <br />
<sub><b>Anand Chowdhary</b></sub> <sub><b>Anand Chowdhary</b></sub>
</a> </a>
@@ -961,7 +914,7 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<a href="https://github.com/shrippen"> <a href="https://github.com/shrippen">
<img src="https://avatars.githubusercontent.com/u/2873570?v=4" width="80;" alt="shrippen"/> <img src="https://avatars.githubusercontent.com/u/2873570?v=4" width="80;" alt="shrippen"/>
<br /> <br />
<sub><b>Shrippen</b></sub> <sub><b>shrippen</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
@@ -973,14 +926,14 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/k-rol"> <a href="https://github.com/k-rol">
<img src="https://avatars.githubusercontent.com/u/4050412?u=1162510eec7b7aeb31d4c7c65d51d4f773d823b0&v=4" width="80;" alt="k-rol"/> <img src="https://avatars.githubusercontent.com/u/4050412?v=4" width="80;" alt="k-rol"/>
<br /> <br />
<sub><b>Carol Ouellet</b></sub> <sub><b>Carol Ouellet</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/bile0026"> <a href="https://github.com/bile0026">
<img src="https://avatars.githubusercontent.com/u/5022496?u=aec96ad173c0ea9baaba93807efa8a848af6595c&v=4" width="80;" alt="bile0026"/> <img src="https://avatars.githubusercontent.com/u/5022496?v=4" width="80;" alt="bile0026"/>
<br /> <br />
<sub><b>Zach Biles</b></sub> <sub><b>Zach Biles</b></sub>
</a> </a>
@@ -988,35 +941,35 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<tr> <tr>
<td align="center"> <td align="center">
<a href="https://github.com/UlisesGascon"> <a href="https://github.com/UlisesGascon">
<img src="https://avatars.githubusercontent.com/u/5110813?u=3c41facd8aa26154b9451de237c34b0f78d672a5&v=4" width="80;" alt="UlisesGascon"/> <img src="https://avatars.githubusercontent.com/u/5110813?v=4" width="80;" alt="UlisesGascon"/>
<br /> <br />
<sub><b>Ulises Gascón</b></sub> <sub><b>Ulises Gascón</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/digitalarche"> <a href="https://github.com/digitalarche">
<img src="https://avatars.githubusercontent.com/u/6546135?u=a1a724d38ee1eba2d2d315495d482256312affe8&v=4" width="80;" alt="digitalarche"/> <img src="https://avatars.githubusercontent.com/u/6546135?v=4" width="80;" alt="digitalarche"/>
<br /> <br />
<sub><b>Digital Archeology</b></sub> <sub><b>Digital Archeology</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/bmcgonag"> <a href="https://github.com/bmcgonag">
<img src="https://avatars.githubusercontent.com/u/7346620?u=2a0f9284f3e12ac1cc15288c254d1ec68a5081e8&v=4" width="80;" alt="bmcgonag"/> <img src="https://avatars.githubusercontent.com/u/7346620?v=4" width="80;" alt="bmcgonag"/>
<br /> <br />
<sub><b>Brian McGonagill</b></sub> <sub><b>Brian McGonagill</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/vlad-timofeev"> <a href="https://github.com/vlad-timofeev">
<img src="https://avatars.githubusercontent.com/u/11474041?u=eee43705b54d2ec9f51fc4fcce5ad18dd17c87e4&v=4" width="80;" alt="vlad-timofeev"/> <img src="https://avatars.githubusercontent.com/u/11474041?v=4" width="80;" alt="vlad-timofeev"/>
<br /> <br />
<sub><b>Vlad Timofeev</b></sub> <sub><b>Vlad Timofeev</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/iJasonWade"> <a href="https://github.com/iJasonWade">
<img src="https://avatars.githubusercontent.com/u/12824479?u=5446c46f50a3197b9cd970e1669ed42e654c66b1&v=4" width="80;" alt="iJasonWade"/> <img src="https://avatars.githubusercontent.com/u/12824479?v=4" width="80;" alt="iJasonWade"/>
<br /> <br />
<sub><b>Jason Ash</b></sub> <sub><b>Jason Ash</b></sub>
</a> </a>
@@ -1031,14 +984,14 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<tr> <tr>
<td align="center"> <td align="center">
<a href="https://github.com/Bastii717"> <a href="https://github.com/Bastii717">
<img src="https://avatars.githubusercontent.com/u/53431819?u=dff9e241e6dbecbc283ee13486a2312cb3dd9784&v=4" width="80;" alt="Bastii717"/> <img src="https://avatars.githubusercontent.com/u/53431819?v=4" width="80;" alt="Bastii717"/>
<br /> <br />
<sub><b>Bastii717</b></sub> <sub><b>Bastii717</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/ratty222"> <a href="https://github.com/ratty222">
<img src="https://avatars.githubusercontent.com/u/92832598?u=137b65530cbd5f5af9c24cde51baa6cc77cc934b&v=4" width="80;" alt="ratty222"/> <img src="https://avatars.githubusercontent.com/u/92832598?v=4" width="80;" alt="ratty222"/>
<br /> <br />
<sub><b>Brent</b></sub> <sub><b>Brent</b></sub>
</a> </a>
@@ -1047,7 +1000,7 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<a href="https://github.com/jtfinley72"> <a href="https://github.com/jtfinley72">
<img src="https://avatars.githubusercontent.com/u/96497997?v=4" width="80;" alt="jtfinley72"/> <img src="https://avatars.githubusercontent.com/u/96497997?v=4" width="80;" alt="jtfinley72"/>
<br /> <br />
<sub><b>Jtfinley72</b></sub> <sub><b>jtfinley72</b></sub>
</a> </a>
</td></tr> </td></tr>
</table> </table>

View File

@@ -1,128 +0,0 @@
name: 🚀 Deploy to AWS
on:
workflow_dispatch:
push:
branches:
- master
tags:
- '*'
paths:
- api/**
- serverless.yml
- package.json
- .github/workflows/deploy-aws.yml
jobs:
deploy-api:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v1
with:
node-version: 16
- name: Cache node_modules
uses: actions/cache@v2
with:
path: node_modules
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Create GitHub deployment for API
uses: chrnorm/deployment-action@releases/v1
id: deployment_api
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
environment: AWS (Backend API)
ref: ${{ github.ref }}
- name: Install Serverless CLI and dependencies
run: |
npm i -g serverless
yarn
- name: Deploy to AWS
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
run: serverless deploy
- name: Update GitHub deployment status (API)
if: always()
uses: chrnorm/deployment-status@releases/v1
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
state: "${{ job.status }}"
deployment_id: ${{ steps.deployment_api.outputs.deployment_id }}
ref: ${{ github.ref }}
deploy-frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v1
with:
node-version: 16
- name: Cache node_modules
uses: actions/cache@v2
with:
path: node_modules
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Create GitHub deployment for Frontend
uses: chrnorm/deployment-action@releases/v1
id: deployment_frontend
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
environment: AWS (Frontend Web UI)
ref: ${{ github.ref }}
- name: Install dependencies and build
run: |
yarn install
yarn build
- name: Setup AWS
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Upload to S3
env:
AWS_S3_BUCKET: 'web-check-frontend'
run: aws s3 sync ./build/ s3://$AWS_S3_BUCKET/ --delete
- name: Invalidate CloudFront cache
uses: chetan/invalidate-cloudfront-action@v2.4
env:
DISTRIBUTION: E30XKAM2TG9FD8
PATHS: '/*'
AWS_REGION: 'us-east-1'
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Update GitHub deployment status (Frontend)
if: always()
uses: chrnorm/deployment-status@releases/v1
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
state: "${{ job.status }}"
deployment_id: ${{ steps.deployment_frontend.outputs.deployment_id }}
ref: ${{ github.ref }}

69
.gitignore vendored
View File

@@ -1,60 +1,27 @@
# ------------------------
# ENVIRONMENT SETTINGS # keys
# ------------------------
.env .env
# ------------------------ # dependencies
# PRODUCTION /node_modules
# ------------------------ /.pnp
/build/ .pnp.js
# ------------------------ # logs
# DEPLOYMENT
# ------------------------
.vercel/
.netlify/
.webpack/
.serverless/
# ------------------------
# DEPENDENCIES
# ------------------------
node_modules/
.yarn/cache/
.yarn/unplugged/
.yarn/build-state.yml
.yarn/install-state.gz
.pnpm/
.pnp.*
# ------------------------
# LOGS
# ------------------------
logs/
*.log
npm-debug.log* npm-debug.log*
yarn-debug.log* yarn-debug.log*
yarn-error.log* yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
# ------------------------ # testing
# TESTING /coverage
# ------------------------
coverage/
.nyc_output/
# ------------------------ # production
# OS SPECIFIC /build
# ------------------------
# Random AWS and Netlify crap
.netlify
.serverless
.webpack
# OS generated files
.DS_Store .DS_Store
Thumbs.db
# ------------------------
# EDITORS
# ------------------------
.idea/
.vscode/
*.swp
*.swo

View File

@@ -1,48 +1,22 @@
# Specify the Node.js version to use FROM node:16-buster-slim
ARG NODE_VERSION=16
# Specify the Debian version to use, the default is "bullseye" RUN apt-get update && \
ARG DEBIAN_VERSION=bullseye apt-get install -y chromium traceroute && \
chmod 755 /usr/bin/chromium && \
rm -rf /var/lib/apt/lists/*
# Use Node.js Docker image as the base image, with specific Node and Debian versions
FROM docker.io/library/node:${NODE_VERSION}-${DEBIAN_VERSION}
# Set the container's default shell to Bash and enable some options
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
# Install Chromium browser and Download and verify Google Chromes signing key
RUN apt-get update -qq --fix-missing && \
apt-get -qqy install --allow-unauthenticated gnupg wget && \
wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google.list && \
apt-get update -qq && \
apt-get -qqy --no-install-recommends install chromium traceroute && \
rm -rf /var/lib/apt/lists/*
# Run the Chromium browser's version command and redirect its output to the /etc/chromium-version file
RUN /usr/bin/chromium --no-sandbox --version > /etc/chromium-version
# Set the working directory to /app
WORKDIR /app WORKDIR /app
# Copy package.json and yarn.lock to the working directory
COPY package.json yarn.lock ./ COPY package.json yarn.lock ./
# Run yarn install to install dependencies and clear yarn cache RUN yarn install
RUN yarn install && rm -rf /app/node_modules/.cache
# Copy all files to working directory
COPY . . COPY . .
RUN mkdir /app/data
# Run yarn build to build the application
RUN yarn build RUN yarn build
# Exposed container port, the default is 3000, which can be modified through the environment variable PORT
EXPOSE ${PORT:-3000} EXPOSE ${PORT:-3000}
# Set the environment variable CHROME_PATH to specify the path to the Chromium binaries
ENV CHROME_PATH='/usr/bin/chromium' ENV CHROME_PATH='/usr/bin/chromium'
# Define the command executed when the container starts and start the server.js of the Node.js application CMD ["yarn", "serve"]
CMD [ "node", "server.js" ]

View File

@@ -2,81 +2,37 @@ const normalizeUrl = (url) => {
return url.startsWith('http') ? url : `https://${url}`; return url.startsWith('http') ? url : `https://${url}`;
}; };
const headers = {
'Access-Control-Allow-Origin': process.env.API_CORS_ORIGIN || '*',
'Access-Control-Allow-Credentials': true,
'Content-Type': 'application/json;charset=UTF-8',
};
const commonMiddleware = (handler) => { const commonMiddleware = (handler) => {
// Vercel return async (event, context, callback) => {
const vercelHandler = async (request, response) => {
const queryParams = request.query || {};
const rawUrl = queryParams.url;
if (!rawUrl) {
return response.status(500).json({ error: 'No URL specified' });
}
const url = normalizeUrl(rawUrl);
try {
const handlerResponse = await handler(url, request);
if (handlerResponse.body && handlerResponse.statusCode) {
response.status(handlerResponse.statusCode).json(handlerResponse.body);
} else {
response.status(200).json(
typeof handlerResponse === 'object' ? handlerResponse : JSON.parse(handlerResponse)
);
}
} catch (error) {
response.status(500).json({ error: error.message });
}
};
// Netlify
const netlifyHandler = async (event, context, callback) => {
const queryParams = event.queryStringParameters || event.query || {}; const queryParams = event.queryStringParameters || event.query || {};
const rawUrl = queryParams.url; const rawUrl = queryParams.url;
if (!rawUrl) { if (!rawUrl) {
callback(null, { callback(null, {
statusCode: 500, statusCode: 500,
body: JSON.stringify({ error: 'No URL specified' }), body: JSON.stringify({ error: 'No URL specified' }),
headers,
}); });
return;
} }
const url = normalizeUrl(rawUrl); const url = normalizeUrl(rawUrl);
try { try {
const handlerResponse = await handler(url, event, context); const response = await handler(url, event, context);
if (handlerResponse.body && handlerResponse.statusCode) { if (response.body && response.statusCode) {
callback(null, handlerResponse); callback(null, response);
} else { } else {
callback(null, { callback(null, {
statusCode: 200, statusCode: 200,
body: typeof handlerResponse === 'object' ? JSON.stringify(handlerResponse) : handlerResponse, body: typeof response === 'object' ? JSON.stringify(response) : response,
headers,
}); });
} }
} catch (error) { } catch (error) {
callback(null, { callback(null, {
statusCode: 500, statusCode: 500,
body: JSON.stringify({ error: error.message }), body: JSON.stringify({ error: error.message }),
headers,
}); });
} }
}; };
// The format of the handlers varies between platforms
// E.g. Netlify + AWS expect Lambda functions, but Vercel or Node needs standard handler
const platformEnv = (process.env.PLATFORM || '').toUpperCase(); // Has user set platform manually?
const nativeMode = (['VERCEL', 'NODE'].includes(platformEnv) || process.env.VERCEL || process.env.WC_SERVER);
return nativeMode ? vercelHandler : netlifyHandler;
}; };
module.exports = commonMiddleware; module.exports = commonMiddleware;

View File

@@ -80,5 +80,4 @@ const getWaybackData = async (url) => {
} }
}; };
module.exports = middleware(getWaybackData); exports.handler = middleware(getWaybackData);
module.exports.handler = middleware(getWaybackData);

View File

@@ -94,12 +94,12 @@ const checkDomainAgainstDnsServers = async (domain) => {
return results; return results;
}; };
const handler = async (url) => { exports.handler = middleware(async (url) => {
const domain = new URL(url).hostname; const domain = new URL(url).hostname;
const results = await checkDomainAgainstDnsServers(domain); const results = await checkDomainAgainstDnsServers(domain);
return { blocklists: results }; return {
}; statusCode: 200,
body: JSON.stringify({ blocklists: results })
module.exports = middleware(handler); };
module.exports.handler = middleware(handler); });

View File

@@ -48,5 +48,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -54,5 +54,4 @@ const handler = async (url) => {
return { headerCookies, clientCookies }; return { headerCookies, clientCookies };
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const dns = require('dns'); const dns = require('dns');
const dnsPromises = dns.promises; const dnsPromises = dns.promises;
const axios = require('axios'); const axios = require('axios');
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const handler = async (url) => { const handler = async (url) => {
try { try {
@@ -41,5 +41,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -51,5 +51,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); // Make sure this path is correct const commonMiddleware = require('./_common/middleware'); // Make sure this path is correct
const handler = async (domain) => { const fetchDNSRecords = async (domain, event, context) => {
const dnsTypes = ['DNSKEY', 'DS', 'RRSIG']; const dnsTypes = ['DNSKEY', 'DS', 'RRSIG'];
const records = {}; const records = {};
@@ -49,6 +49,4 @@ const handler = async (domain) => {
return records; return records;
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(fetchDNSRecords);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const builtWithHandler = async (url) => {
const apiKey = process.env.BUILT_WITH_API_KEY; const apiKey = process.env.BUILT_WITH_API_KEY;
if (!url) { if (!url) {
@@ -45,5 +45,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(builtWithHandler);
module.exports.handler = middleware(handler);

View File

@@ -102,5 +102,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -18,6 +18,4 @@ const handler = async (url) => {
return await lookupAsync(address); return await lookupAsync(address);
}; };
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -15,5 +15,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,15 +1,18 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url, event, context) => { exports.handler = middleware(async (url, event, context) => {
const errorResponse = (message, statusCode = 500) => { const errorResponse = (message, statusCode = 500) => {
return { return {
statusCode: statusCode, statusCode: statusCode,
body: JSON.stringify({ error: message }), body: JSON.stringify({ error: message }),
}; };
}; };
const hstsIncompatible = (message, compatible = false, hstsHeader = null ) => { const hstsIncompatible = (message, statusCode = 200) => {
return { message, compatible, hstsHeader }; return {
statusCode: statusCode,
body: JSON.stringify({ message, compatible: false }),
};
}; };
@@ -32,7 +35,14 @@ const handler = async (url, event, context) => {
} else if (!preload) { } else if (!preload) {
resolve(hstsIncompatible(`HSTS header does not contain the preload directive.`)); resolve(hstsIncompatible(`HSTS header does not contain the preload directive.`));
} else { } else {
resolve(hstsIncompatible(`Site is compatible with the HSTS preload list!`, true, hstsHeader)); resolve({
statusCode: 200,
body: JSON.stringify({
message: "Site is compatible with the HSTS preload list!",
compatible: true,
hstsHeader: hstsHeader,
}),
});
} }
} }
}); });
@@ -43,8 +53,5 @@ const handler = async (url, event, context) => {
req.end(); req.end();
}); });
}; });
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -8,7 +8,7 @@ const handler = async (url) => {
const response = await axios.get(fullUrl); const response = await axios.get(fullUrl);
const headers = response.headers; const headers = response.headers;
return { return {
strictTransportPolicy: headers['strict-transport-security'] ? true : false, strictTransportPolicy: headers['strict-transport-policy'] ? true : false,
xFrameOptions: headers['x-frame-options'] ? true : false, xFrameOptions: headers['x-frame-options'] ? true : false,
xContentTypeOptions: headers['x-content-type-options'] ? true : false, xContentTypeOptions: headers['x-content-type-options'] ? true : false,
xXSSProtection: headers['x-xss-protection'] ? true : false, xXSSProtection: headers['x-xss-protection'] ? true : false,
@@ -22,5 +22,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -66,6 +66,5 @@ return new Promise((resolve, reject) => {
}); });
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const axios = require('axios'); const axios = require('axios');
const cheerio = require('cheerio'); const cheerio = require('cheerio');
const urlLib = require('url'); const urlLib = require('url');
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const handler = async (url) => { const handler = async (url) => {
const response = await axios.get(url); const response = await axios.get(url);
@@ -45,5 +45,4 @@ const handler = async (url) => {
return { internal: internalLinks, external: externalLinks }; return { internal: internalLinks, external: externalLinks };
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const dns = require('dns').promises; const dns = require('dns').promises;
const URL = require('url-parse'); const URL = require('url-parse');
@@ -72,5 +72,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); module.exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -73,12 +73,17 @@ const handler = async (url, event, context) => {
return errorResponse('The function timed out before completing.'); return errorResponse('The function timed out before completing.');
} }
return { openPorts, failedPorts }; return {
statusCode: 200,
body: JSON.stringify({ openPorts, failedPorts }),
};
}; };
const errorResponse = (message, statusCode = 444) => { const errorResponse = (message, statusCode = 444) => {
return { error: message }; return {
statusCode: statusCode,
body: JSON.stringify({ error: message }),
};
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -4,6 +4,10 @@ const middleware = require('./_common/middleware');
const handler = async (url, event, context) => { const handler = async (url, event, context) => {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY; const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!url) {
throw new Error('URL param is required');
}
if (!apiKey) { if (!apiKey) {
throw new Error('API key (GOOGLE_CLOUD_API_KEY) not set'); throw new Error('API key (GOOGLE_CLOUD_API_KEY) not set');
} }
@@ -15,5 +19,4 @@ const handler = async (url, event, context) => {
return response.data; return response.data;
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -10,7 +10,7 @@ const handler = async (url) => {
{ auth: { username: process.env.TRANCO_USERNAME, password: process.env.TRANCO_API_KEY } } { auth: { username: process.env.TRANCO_USERNAME, password: process.env.TRANCO_API_KEY } }
: {}; : {};
const response = await axios.get( const response = await axios.get(
`https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 5000 }, auth, `https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 2000 }, auth,
); );
if (!response.data || !response.data.ranks || response.data.ranks.length === 0) { if (!response.data || !response.data.ranks || response.data.ranks.length === 0) {
return { skipped: `Skipping, as ${domain} isn't ranked in the top 100 million sites yet.`}; return { skipped: `Skipping, as ${domain} isn't ranked in the top 100 million sites yet.`};
@@ -21,6 +21,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -24,6 +24,4 @@ const handler = async (url) => {
}; };
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -67,5 +67,4 @@ const handler = async function(url) {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const puppeteer = require('puppeteer-core');
const chromium = require('chrome-aws-lambda'); const chromium = require('chrome-aws-lambda');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (targetUrl) => { const screenshotHandler = async (targetUrl) => {
if (!targetUrl) { if (!targetUrl) {
throw new Error('URL is missing from queryStringParameters'); throw new Error('URL is missing from queryStringParameters');
@@ -21,7 +21,7 @@ const handler = async (targetUrl) => {
let browser = null; let browser = null;
try { try {
browser = await puppeteer.launch({ browser = await puppeteer.launch({
args: [...chromium.args, '--no-sandbox'], // Add --no-sandbox flag args: chromium.args,
defaultViewport: { width: 800, height: 600 }, defaultViewport: { width: 800, height: 600 },
executablePath: process.env.CHROME_PATH || await chromium.executablePath, executablePath: process.env.CHROME_PATH || await chromium.executablePath,
headless: chromium.headless, headless: chromium.headless,
@@ -58,5 +58,4 @@ const handler = async (targetUrl) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(screenshotHandler);
module.exports.handler = middleware(handler);

View File

@@ -38,7 +38,7 @@ const isPgpSigned = (result) => {
return false; return false;
}; };
const handler = async (urlParam) => { const securityTxtHandler = async (urlParam) => {
let url; let url;
try { try {
@@ -69,6 +69,8 @@ const handler = async (urlParam) => {
return { isPresent: false }; return { isPresent: false };
}; };
exports.handler = middleware(securityTxtHandler);
async function fetchSecurityTxt(baseURL, path) { async function fetchSecurityTxt(baseURL, path) {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const url = new URL(path, baseURL); const url = new URL(path, baseURL);
@@ -89,6 +91,3 @@ async function fetchSecurityTxt(baseURL, path) {
}); });
}); });
} }
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const axios = require('axios'); const axios = require('axios');
const xml2js = require('xml2js'); const xml2js = require('xml2js');
@@ -25,7 +25,10 @@ const handler = async (url) => {
} }
if (!sitemapUrl) { if (!sitemapUrl) {
return { skipped: 'No sitemap found' }; return {
statusCode: 404,
body: JSON.stringify({ skipped: 'No sitemap found' }),
};
} }
sitemapRes = await axios.get(sitemapUrl, { timeout: 5000 }); sitemapRes = await axios.get(sitemapUrl, { timeout: 5000 });
@@ -37,18 +40,25 @@ const handler = async (url) => {
const parser = new xml2js.Parser(); const parser = new xml2js.Parser();
const sitemap = await parser.parseStringPromise(sitemapRes.data); const sitemap = await parser.parseStringPromise(sitemapRes.data);
return sitemap; return {
statusCode: 200,
body: JSON.stringify(sitemap),
};
} catch (error) { } catch (error) {
// If error occurs // If error occurs
console.log(error.message); console.log(error.message);
if (error.code === 'ECONNABORTED') { if (error.code === 'ECONNABORTED') {
return { error: 'Request timed out' }; return {
statusCode: 500,
body: JSON.stringify({ error: 'Request timed out' }),
};
} else { } else {
return { error: error.message }; return {
statusCode: 500,
body: JSON.stringify({ error: error.message }),
};
} }
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const axios = require('axios'); const axios = require('axios');
const cheerio = require('cheerio'); const cheerio = require('cheerio');
@@ -50,9 +50,16 @@ const handler = async (url) => {
}; };
if (Object.keys(metadata).length === 0) { if (Object.keys(metadata).length === 0) {
return { skipped: 'No metadata found' }; return {
statusCode: 200,
body: JSON.stringify({ skipped: 'No metadata found' }),
};
} }
return metadata;
return {
statusCode: 200,
body: JSON.stringify(metadata),
};
} catch (error) { } catch (error) {
return { return {
statusCode: 500, statusCode: 500,
@@ -61,5 +68,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const tls = require('tls'); const tls = require('tls');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (urlString) => { const fetchSiteCertificateHandler = async (urlString) => {
try { try {
const parsedUrl = new URL(urlString); const parsedUrl = new URL(urlString);
const options = { const options = {
@@ -40,5 +40,4 @@ const handler = async (urlString) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(fetchSiteCertificateHandler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const https = require('https');
const { performance, PerformanceObserver } = require('perf_hooks'); const { performance, PerformanceObserver } = require('perf_hooks');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const checkURLHandler = async (url) => {
if (!url) { if (!url) {
throw new Error('You must provide a URL query parameter!'); throw new Error('You must provide a URL query parameter!');
} }
@@ -55,5 +55,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(checkURLHandler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const Wappalyzer = require('wappalyzer'); const Wappalyzer = require('wappalyzer');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const analyzeSiteTechnologies = async (url) => {
const options = {}; const options = {};
const wappalyzer = new Wappalyzer(options); const wappalyzer = new Wappalyzer(options);
@@ -27,5 +27,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(analyzeSiteTechnologies);
module.exports.handler = middleware(handler);

View File

@@ -5,9 +5,6 @@ const middleware = require('./_common/middleware');
const getGoogleSafeBrowsingResult = async (url) => { const getGoogleSafeBrowsingResult = async (url) => {
try { try {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY; const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!apiKey) {
return { error: 'GOOGLE_CLOUD_API_KEY is required for the Google Safe Browsing check' };
}
const apiEndpoint = `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`; const apiEndpoint = `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`;
const requestBody = { const requestBody = {
@@ -66,15 +63,11 @@ const getPhishTankResult = async (url) => {
} }
const getCloudmersiveResult = async (url) => { const getCloudmersiveResult = async (url) => {
const apiKey = process.env.CLOUDMERSIVE_API_KEY;
if (!apiKey) {
return { error: 'CLOUDMERSIVE_API_KEY is required for the Cloudmersive check' };
}
try { try {
const endpoint = 'https://api.cloudmersive.com/virus/scan/website'; const endpoint = 'https://api.cloudmersive.com/virus/scan/website';
const headers = { const headers = {
'Content-Type': 'application/x-www-form-urlencoded', 'Content-Type': 'application/x-www-form-urlencoded',
'Apikey': apiKey, 'Apikey': process.env.CLOUDMERSIVE_API_KEY,
}; };
const data = `Url=${encodeURIComponent(url)}`; const data = `Url=${encodeURIComponent(url)}`;
const response = await axios.post(endpoint, data, { headers }); const response = await axios.post(endpoint, data, { headers });
@@ -99,5 +92,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -25,5 +25,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const traceroute = require('traceroute');
const url = require('url'); const url = require('url');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (urlString, context) => { const executeTraceroute = async (urlString, context) => {
// Parse the URL and get the hostname // Parse the URL and get the hostname
const urlObject = url.parse(urlString); const urlObject = url.parse(urlString);
const host = urlObject.hostname; const host = urlObject.hostname;
@@ -28,5 +28,4 @@ const handler = async (urlString, context) => {
}; };
}; };
module.exports = middleware(handler); exports.handler = middleware(executeTraceroute);
module.exports.handler = middleware(handler);

View File

@@ -29,5 +29,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -83,7 +83,7 @@ const fetchFromMyAPI = async (hostname) => {
} }
}; };
const handler = async (url) => { const fetchWhoisData = async (url) => {
if (!url.startsWith('http://') && !url.startsWith('https://')) { if (!url.startsWith('http://') && !url.startsWith('https://')) {
url = 'http://' + url; url = 'http://' + url;
} }
@@ -106,6 +106,4 @@ const handler = async (url) => {
}; };
}; };
module.exports = middleware(handler); exports.handler = middleware(fetchWhoisData);
module.exports.handler = middleware(handler);

View File

@@ -4,7 +4,7 @@
"private": false, "private": false,
"description": "All-in-one OSINT tool for analyzing any website", "description": "All-in-one OSINT tool for analyzing any website",
"repository": "github:lissy93/web-check", "repository": "github:lissy93/web-check",
"homepage": "https://web-check.xyz", "homepage": "https://web-check.as93.net",
"license": "MIT", "license": "MIT",
"author": { "author": {
"name": "Alicia Sykes", "name": "Alicia Sykes",
@@ -20,9 +20,7 @@
"start": "react-scripts start", "start": "react-scripts start",
"build": "react-scripts build", "build": "react-scripts build",
"test": "react-scripts test", "test": "react-scripts test",
"dev:vercel": "PLATFORM='vercel' npx vercel dev", "eject": "react-scripts eject"
"dev:netlify": "PLATFORM='netlify' npx netlify dev",
"dev:node": "npx concurrently --names \"frontend,backend\" \"REACT_APP_API_ENDPOINT='http://localhost:3001/api' npx react-scripts start\" \"PLATFORM='node' DISABLE_GUI='true' PORT='3001' API_CORS_ORIGIN='*' npx nodemon server\""
}, },
"dependencies": { "dependencies": {
"@netlify/functions": "^1.6.0", "@netlify/functions": "^1.6.0",
@@ -37,12 +35,12 @@
"@types/react-router-dom": "^5.3.3", "@types/react-router-dom": "^5.3.3",
"@types/react-simple-maps": "^3.0.0", "@types/react-simple-maps": "^3.0.0",
"@types/styled-components": "^5.1.26", "@types/styled-components": "^5.1.26",
"aws-serverless-express": "^3.4.0",
"axios": "^1.4.0", "axios": "^1.4.0",
"cheerio": "^1.0.0-rc.12", "cheerio": "^1.0.0-rc.12",
"chrome-aws-lambda": "^10.1.0", "chrome-aws-lambda": "^10.1.0",
"chromium": "^3.0.3", "chromium": "^3.0.3",
"connect-history-api-fallback": "^2.0.0", "connect-history-api-fallback": "^2.0.0",
"cors": "^2.8.5",
"csv-parser": "^3.0.0", "csv-parser": "^3.0.0",
"dotenv": "^16.3.1", "dotenv": "^16.3.1",
"flatted": "^3.2.7", "flatted": "^3.2.7",

View File

@@ -1,39 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p class="moji">😪</p>
<p>There was an error finding this route.</p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
.moji { font-size: 8rem; margin: 0; }
</style>
</body>
</html>

View File

@@ -18,7 +18,7 @@
<meta property="og:title" content="Web Check"> <meta property="og:title" content="Web Check">
<meta property="og:description" content="All-in-one Website OSINT Scanner"> <meta property="og:description" content="All-in-one Website OSINT Scanner">
<meta property="og:image" content="/banner.png"> <meta property="og:image" content="/banner.png">
<meta property="og:url" content="https://web-check.xyz"> <meta property="og:url" content="https://web-check.as93.net">
<meta name="twitter:card" content="summary_large_image"> <meta name="twitter:card" content="summary_large_image">
</head> </head>
<body> <body>

View File

@@ -1,37 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p><!-- CONTENT --></p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
</style>
</body>
</html>

190
server.js
View File

@@ -1,148 +1,116 @@
const express = require('express'); const express = require('express');
const awsServerlessExpress = require('aws-serverless-express');
const fs = require('fs'); const fs = require('fs');
const path = require('path'); const path = require('path');
const cors = require('cors');
const historyApiFallback = require('connect-history-api-fallback'); const historyApiFallback = require('connect-history-api-fallback');
require('dotenv').config(); require('dotenv').config();
const app = express(); const app = express();
const port = process.env.PORT || 3000; // The port to run the server on
const API_DIR = '/api'; // Name of the dir containing the lambda functions const API_DIR = '/api'; // Name of the dir containing the lambda functions
const dirPath = path.join(__dirname, API_DIR); // Path to the lambda functions dir const dirPath = path.join(__dirname, API_DIR); // Path to the lambda functions dir
const guiPath = path.join(__dirname, 'build'); const guiPath = path.join(__dirname, 'build');
const placeholderFilePath = path.join(__dirname, 'public', 'placeholder.html');
const handlers = {}; // Will store list of API endpoints
process.env.WC_SERVER = 'true'; // Tells middleware to return in non-lambda mode
// Enable CORS // Execute the lambda function
app.use(cors({ const executeHandler = async (handler, req) => {
origin: process.env.API_CORS_ORIGIN || '*', return new Promise((resolve, reject) => {
})); const callback = (err, response) => err ? reject(err) : resolve(response);
const promise = handler(req, {}, callback);
// Read and register each API function as an Express routes if (promise && typeof promise.then === 'function') {
fs.readdirSync(dirPath, { withFileTypes: true }) promise.then(resolve).catch(reject);
}
});
};
// Array of all the lambda function file names
const fileNames = fs.readdirSync(dirPath, { withFileTypes: true })
.filter(dirent => dirent.isFile() && dirent.name.endsWith('.js')) .filter(dirent => dirent.isFile() && dirent.name.endsWith('.js'))
.forEach(dirent => { .map(dirent => dirent.name);
const routeName = dirent.name.split('.')[0];
const route = `${API_DIR}/${routeName}`;
const handler = require(path.join(dirPath, dirent.name));
handlers[route] = handler;
app.get(route, async (req, res) => { const handlers = {};
try {
await handler(req, res); fileNames.forEach(file => {
} catch (err) { const routeName = file.split('.')[0];
res.status(500).json({ error: err.message }); const route = `${API_DIR}/${routeName}`;
} const handler = require(path.join(dirPath, file)).handler;
});
handlers[route] = handler;
app.get(route, async (req, res) => {
try {
const { statusCode = 200, body = '' } = await executeHandler(handler, req);
res.status(statusCode).json(JSON.parse(body));
} catch (err) {
res.status(500).json({ error: err.message });
}
});
});
const timeout = (ms, jobName = null) => {
return new Promise((_, reject) => {
setTimeout(() => {
reject(new Error(`Timed out after the ${ms/1000} second limit${jobName ? `, when executing the ${jobName} task` : ''}`));
}, ms);
});
}
app.get('/api', async (req, res) => {
const results = {};
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 15000;
const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => {
const routeName = route.replace(`${API_DIR}/`, '');
try {
const result = await Promise.race([
executeHandler(handler, { query: { url } }),
timeout(maxExecutionTime, routeName)
]);
results[routeName] = JSON.parse((result || {}).body);
} catch (err) {
results[routeName] = { error: err.message };
}
}); });
// Create a single API endpoint to execute all lambda functions await Promise.all(handlerPromises);
app.get('/api', async (req, res) => { res.json(results);
const results = {}; });
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 20000;
const executeHandler = async (handler, req, res) => {
return new Promise(async (resolve, reject) => {
try {
const mockRes = {
status: (statusCode) => mockRes,
json: (body) => resolve({ body }),
};
await handler({ ...req, query: { url } }, mockRes);
} catch (err) {
reject(err);
}
});
};
const timeout = (ms, jobName = null) => {
return new Promise((_, reject) => {
setTimeout(() => {
reject(new Error(
`Timed out after ${ms/1000} seconds${jobName ? `, when executing ${jobName}` : ''}`
));
}, ms);
});
};
const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => {
const routeName = route.replace(`${API_DIR}/`, '');
try {
const result = await Promise.race([
executeHandler(handler, req, res),
timeout(maxExecutionTime, routeName)
]);
results[routeName] = result.body;
} catch (err) {
results[routeName] = { error: err.message };
}
});
await Promise.all(handlerPromises);
res.json(results);
});
// Handle SPA routing // Handle SPA routing
app.use(historyApiFallback({ app.use(historyApiFallback({
rewrites: [ rewrites: [
{ from: /^\/api\/.*$/, to: (context) => context.parsedUrl.path }, { from: /^\/api\/.*$/, to: function(context) { return context.parsedUrl.path; } },
] ]
})); }));
// Serve up the GUI - if build dir exists, and GUI feature enabled // Serve up the GUI - if build dir exists, and GUI feature enabled
if (process.env.DISABLE_GUI && process.env.DISABLE_GUI !== 'false') { if (process.env.DISABLE_GUI && process.env.DISABLE_GUI !== 'false') {
app.get('*', async (req, res) => { app.get('*', (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8'); res.status(500).send(
const htmlContent = placeholderContent.replace( 'Welcome to Web-Check!<br />Access the API endpoints at '
'<!-- CONTENT -->', +'<a href="/api"><code>/api</code></a>'
'Web-Check API is up and running!<br />Access the endpoints at ' );
+'<a href="/api"><code>/api</code></a>'
);
res.status(500).send(htmlContent);
}); });
} else if (!fs.existsSync(guiPath)) { } else if (!fs.existsSync(guiPath)) {
app.get('*', async (req, res) => { app.get('*', (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8'); res.status(500).send(
const htmlContent = placeholderContent.replace( 'Welcome to Web-Check!<br />Looks like the GUI app has not yet been compiled, '
'<!-- CONTENT -->', +'run <code>yarn build</code> to continue, then restart the server.'
'Looks like the GUI app has not yet been compiled.<br /> ' +
'Run <code>yarn build</code> to continue, then restart the server.'
); );
res.status(500).send(htmlContent);
}); });
} else { // GUI enabled, and build files present, let's go!! } else { // GUI enabled, and build files present, let's go!!
app.use(express.static(guiPath)); app.use(express.static(guiPath));
} }
app.use((req, res, next) => { // Create serverless express server
res.status(404).sendFile(path.join(__dirname, 'public', 'error.html')); const port = process.env.PORT || 3000;
const server = awsServerlessExpress.createServer(app).listen(port, () => {
console.log(`Server is running on port ${port}`);
}); });
// Print nice welcome message to user exports.handler = (event, context) => {
const printMessage = () => { awsServerlessExpress.proxy(server, event, context);
console.log(
`\x1b[36m\n` +
' __ __ _ ___ _ _ \n' +
' \\ \\ / /__| |__ ___ / __| |_ ___ __| |__\n' +
' \\ \\/\\/ / -_) \'_ \\___| (__| \' \\/ -_) _| / /\n' +
' \\_/\\_/\\___|_.__/ \\___|_||_\\___\\__|_\\_\\\n' +
`\x1b[0m\n`,
`\x1b[1m\x1b[32m🚀 Web-Check is up and running at http://localhost:${port} \x1b[0m\n\n`,
`\x1b[2m\x1b[36m🛟 For documentation and support, visit the GitHub repo: ` +
`https://github.com/lissy93/web-check \n`,
`💖 Found Web-Check useful? Consider sponsoring us on GitHub ` +
`to help fund maintenance & development.\x1b[0m`
);
}; };
// Create server
app.listen(port, () => {
printMessage();
});

View File

@@ -4,214 +4,121 @@ provider:
name: aws name: aws
runtime: nodejs14.x runtime: nodejs14.x
region: us-east-1 region: us-east-1
# environment:
# GOOGLE_CLOUD_API_KEY: ${ssm:GOOGLE_CLOUD_API_KEY~true, ''}
# TORRENT_IP_API_KEY: ${ssm:TORRENT_IP_API_KEY~true, ''}
# SECURITY_TRAILS_API_KEY: ${ssm:SECURITY_TRAILS_API_KEY~true, ''}
# BUILT_WITH_API_KEY: ${ssm:BUILT_WITH_API_KEY~true, ''}
# URL_SCAN_API_KEY: ${ssm:URL_SCAN_API_KEY~true, ''}
# TRANCO_USERNAME: ${ssm:TRANCO_USERNAME~true, ''}
# TRANCO_API_KEY: ${ssm:TRANCO_API_KEY~true, ''}
# CLOUDMERSIVE_API_KEY: ${ssm:CLOUDMERSIVE_API_KEY~true, ''}
# CHROME_PATH: ${ssm:CHROME_PATH~true, ''}
# API_TIMEOUT_LIMIT: ${ssm:API_TIMEOUT_LIMIT~true, ''}
# API_CORS_ORIGIN: ${ssm:API_CORS_ORIGIN~true, ''}
iamRoleStatements:
- Effect: Allow
Action:
- ssm:GetParameter
Resource:
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/GOOGLE_CLOUD_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TORRENT_IP_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/SECURITY_TRAILS_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/BUILT_WITH_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/URL_SCAN_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TRANCO_USERNAME
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TRANCO_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/CLOUDMERSIVE_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/CHROME_PATH
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/API_TIMEOUT_LIMIT
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/API_CORS_ORIGIN
functions: functions:
archives:
handler: api/archives.handler
events:
- http:
path: api/archives
method: get
blockLists:
handler: api/block-lists.handler
events:
- http:
path: api/block-lists
method: get
carbon:
handler: api/carbon.handler
events:
- http:
path: api/carbon
method: get
cookies:
handler: api/cookies.handler
events:
- http:
path: api/cookies
method: get
dnsServer:
handler: api/dns-server.handler
events:
- http:
path: api/dns-server
method: get
dns:
handler: api/dns.handler
events:
- http:
path: api/dns
method: get
dnssec: dnssec:
handler: api/dnssec.handler handler: api/dnssec.handler
events: events:
- http: - http:
path: api/dnssec path: api/dnssec
method: get method: get
features:
handler: api/features.handler
events:
- http:
path: api/features
method: get
firewall:
handler: api/firewall.handler
events:
- http:
path: api/firewall
method: get
getIp:
handler: api/get-ip.handler
events:
- http:
path: api/get-ip
method: get
headers:
handler: api/headers.handler
events:
- http:
path: api/headers
method: get
hsts:
handler: api/hsts.handler
events:
- http:
path: api/hsts
method: get
httpSecurity:
handler: api/http-security.handler
events:
- http:
path: api/http-security
method: get
legacyRank:
handler: api/legacy-rank.handler
events:
- http:
path: api/legacy-rank
method: get
linkedPages: linkedPages:
handler: api/linked-pages.handler handler: api/linked-pages.handler
events: events:
- http: - http:
path: api/linked-pages path: api/linked-pages
method: get method: get
mailConfig:
handler: api/mail-config.handler
events:
- http:
path: api/mail-config
method: get
ports:
handler: api/ports.handler
events:
- http:
path: api/ports
method: get
quality:
handler: api/quality.handler
events:
- http:
path: api/quality
method: get
rank:
handler: api/rank.handler
events:
- http:
path: api/rank
method: get
redirects:
handler: api/redirects.handler
events:
- http:
path: api/redirects
method: get
robotsTxt: robotsTxt:
handler: api/robots-txt.handler handler: api/robots-txt.handler
events: events:
- http: - http:
path: api/robots-txt path: api/robots-txt
method: get method: get
screenshot:
handler: api/screenshot.handler
events:
- http:
path: api/screenshot
method: get
securityTxt:
handler: api/security-txt.handler
events:
- http:
path: api/security-txt
method: get
sitemap:
handler: api/sitemap.handler
events:
- http:
path: api/sitemap
method: get
socialTags:
handler: api/social-tags.handler
events:
- http:
path: api/social-tags
method: get
ssl: ssl:
handler: api/ssl.handler handler: api/ssl.handler
events: events:
- http: - http:
path: api/ssl path: api/ssl
method: get method: get
whois:
handler: api/whois.handler
events:
- http:
path: api/whois
method: get
carbon:
handler: api/carbon.handler
events:
- http:
path: api/carbon
method: get
features:
handler: api/features.handler
events:
- http:
path: api/features
method: get
mailConfig:
handler: api/mail-config.handler
events:
- http:
path: api/mail-config
method: get
screenshot:
handler: api/screenshot.handler
events:
- http:
path: api/screenshot
method: get
status: status:
handler: api/status.handler handler: api/status.handler
events: events:
- http: - http:
path: api/status path: api/status
method: get method: get
cookies:
handler: api/cookies.handler
events:
- http:
path: api/cookies
method: get
getIp:
handler: api/get-ip.handler
events:
- http:
path: api/get-ip
method: get
ports:
handler: api/ports.handler
events:
- http:
path: api/ports
method: get
securityTxt:
handler: api/security-txt.handler
events:
- http:
path: api/security-txt
method: get
techStack: techStack:
handler: api/tech-stack.handler handler: api/tech-stack.handler
events: events:
- http: - http:
path: api/tech-stack path: api/tech-stack
method: get method: get
threats: dnsServer:
handler: api/threats.handler handler: api/dns-server.handler
events: events:
- http: - http:
path: api/threats path: api/dns-server
method: get method: get
tls: headers:
handler: api/tls.handler handler: api/headers.handler
events: events:
- http: - http:
path: api/tls path: api/headers
method: get
quality:
handler: api/quality.handler
events:
- http:
path: api/quality
method: get
sitemap:
handler: api/sitemap.handler
events:
- http:
path: api/sitemap
method: get method: get
traceRoute: traceRoute:
handler: api/trace-route.handler handler: api/trace-route.handler
@@ -219,38 +126,54 @@ functions:
- http: - http:
path: api/trace-route path: api/trace-route
method: get method: get
dns:
handler: api/dns.handler
events:
- http:
path: api/dns
method: get
hsts:
handler: api/hsts.handler
events:
- http:
path: api/hsts
method: get
redirects:
handler: api/redirects.handler
events:
- http:
path: api/redirects
method: get
socialTags:
handler: api/social-tags.handler
events:
- http:
path: api/social-tags
method: get
txtRecords: txtRecords:
handler: api/txt-records.handler handler: api/txt-records.handler
events: events:
- http: - http:
path: api/txt-records path: api/txt-records
method: get method: get
whois:
handler: api/whois.handler
events:
- http:
path: api/whois
method: get
plugins: plugins:
- serverless-webpack # - serverless-webpack
# - serverless-domain-manager - serverless-domain-manager
# - serverless-offline - serverless-offline
custom: custom:
webpack: webpack:
webpackConfig: 'api/_common/aws-webpack.config.js' webpackConfig: 'api/_common/aws-webpack.config.js'
includeModules: false includeModules: true
packagerOptions:
noInstall: true
# customDomain: customDomain:
# domainName: example.com domainName: example.com
# basePath: 'api' basePath: 'api'
# stage: ${self:provider.stage} stage: ${self:provider.stage}
# createRoute53Record: true createRoute53Record: true
# serverless-offline: serverless-offline:
# prefix: '' prefix: ''
# httpPort: 3000 httpPort: 3000

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
import Button from 'components/Form/Button'; import Button from 'components/Form/Button';
import { ExpandableRow } from 'components/Form/Row'; import { ExpandableRow } from 'components/Form/Row';
@@ -29,10 +29,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [cipherSuites, setCipherSuites] = useState(makeCipherSuites(props.data)); const [cipherSuites, setCipherSuites] = useState(makeCipherSuites(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => { // Update cipher suites when data changes
setCipherSuites(makeCipherSuites(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setCipherSuites([]); setCipherSuites([]);
setLoadState('loading'); setLoadState('loading');

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
import Button from 'components/Form/Button'; import Button from 'components/Form/Button';
import { ExpandableRow } from 'components/Form/Row'; import { ExpandableRow } from 'components/Form/Row';
@@ -38,10 +38,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [clientSupport, setClientSupport] = useState(makeClientSupport(props.data)); const [clientSupport, setClientSupport] = useState(makeClientSupport(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => {
setClientSupport(makeClientSupport(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setClientSupport([]); setClientSupport([]);
setLoadState('loading'); setLoadState('loading');

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import styled from 'styled-components'; import styled from 'styled-components';
import colors from 'styles/colors'; import colors from 'styles/colors';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
@@ -74,11 +74,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [tlsResults, setTlsResults] = useState(makeResults(props.data)); const [tlsResults, setTlsResults] = useState(makeResults(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => {
setTlsRowWata(makeExpandableData(props.data));
setTlsResults(makeResults(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setTlsRowWata([]); setTlsRowWata([]);
setLoadState('loading'); setLoadState('loading');

View File

@@ -179,79 +179,6 @@ const About = (): JSX.Element => {
))} ))}
</Section> </Section>
<Heading as="h2" size="medium" color={colors.primary}>Deploy your own Instance</Heading>
<Section>
<p>Web-Check is designed to be easily self-hosted.</p>
<Heading as="h3" size="small" color={colors.primary}>Option #1 - Netlify</Heading>
<p>Click the button below to deploy to Netlify</p>
<a href="https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check">
<img src="https://www.netlify.com/img/deploy/button.svg" alt="Deploy to Netlify" />
</a>
<Heading as="h3" size="small" color={colors.primary}>Option #2 - Vercel</Heading>
<p>Click the button below to deploy to Vercel</p>
<a href="https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flissy93%2Fweb-check&project-name=web-check&repository-name=web-check-fork&demo-title=Web-Check%20Demo&demo-description=Check%20out%20web-check.xyz%20to%20see%20a%20live%20demo%20of%20this%20application%20running.&demo-url=https%3A%2F%2Fweb-check.xyz&demo-image=https%3A%2F%2Fraw.githubusercontent.com%2FLissy93%2Fweb-check%2Fmaster%2F.github%2Fscreenshots%2Fweb-check-screenshot10.png">
<img src="https://vercel.com/button" alt="Deploy with Vercel" />
</a>
<Heading as="h3" size="small" color={colors.primary}>Option #3 - Docker</Heading>
<p>
A Docker container is published to <a href="https://hub.docker.com/r/lissy93/web-check">DockerHub</a>
<br />
Run this command, then open <code>localhost:3000</code>
<pre>docker run -p 3000:3000 lissy93/web-check</pre>
</p>
<Heading as="h3" size="small" color={colors.primary}>Option #4 - Manual</Heading>
<pre>
git clone https://github.com/Lissy93/web-check.git<br />
cd web-check # Move into the project directory<br />
yarn install # Install dependencies<br />
yarn build # Build the app for production<br />
yarn serve # Start the app (API and GUI)<br />
</pre>
<Heading as="h3" size="small" color={colors.primary}>Further Docs</Heading>
<p>
More detailed installation and setup instructions can be found in the
GitHub repository - <a href="https://github.com/lissy93/web-check#readme">github.com/lissy93/web-check</a>
</p>
<Heading as="h3" size="small" color={colors.primary}>Configuring</Heading>
<p>
There are some optional environmental variables you can specify to give you access to some additional Web-Checks.
See the README for full list of options.
</p>
<ul>
<li>
<code>GOOGLE_CLOUD_API_KEY</code>
: <a href="https://cloud.google.com/api-gateway/docs/authenticate-api-keys">A Google API key</a>
<i> Used to return quality metrics for a site</i>
</li>
<li>
<code>REACT_APP_SHODAN_API_KEY</code>
: <a href="https://account.shodan.io/">A Shodan API key</a>
<i> To show associated hosts for a domain</i>
</li>
<li>
<code>REACT_APP_WHO_API_KEY</code>
: <a href="https://whoapi.com/">A WhoAPI key</a>
<i> Allows for more comprehensive WhoIs records</i>
</li>
</ul>
{/*
**Configuration Settings**:
- `CHROME_PATH` (e.g. `/usr/bin/chromium`) - The path the the Chromium executable
- `PORT` (e.g. `3000`) - Port to serve the API, when running server.js
- `DISABLE_GUI` (e.g. `false`) - Disable the GUI, and only serve the API
- `API_TIMEOUT_LIMIT` (e.g. `10000`) - The timeout limit for API requests, in milliseconds
- `REACT_APP_API_ENDPOINT` (e.g. `/api`) - The endpoint for the API (can be local or remote)</p> */}
</Section>
<Heading as="h2" size="medium" color={colors.primary}>API Documentation</Heading> <Heading as="h2" size="medium" color={colors.primary}>API Documentation</Heading>
<Section> <Section>
{/* eslint-disable-next-line*/} {/* eslint-disable-next-line*/}
@@ -291,6 +218,18 @@ const About = (): JSX.Element => {
Neither your IP address, browser/OS/hardware info, nor any other data will ever be collected or logged. Neither your IP address, browser/OS/hardware info, nor any other data will ever be collected or logged.
(You may verify this yourself, either by inspecting the source code or the using developer tools) (You may verify this yourself, either by inspecting the source code or the using developer tools)
</p> </p>
<hr />
<Heading as="h3" size="small" color={colors.primary}>Support</Heading>
<p>
If you've found something that doesn't work as expected, or would like to ask any questions,
you can open a ticket at <a href="https://github.com/lissy93/web-check/issues">github.com/lissy93/web-check/issues</a>
</p>
<hr />
<Heading as="h3" size="small" color={colors.primary}>Sponsor</Heading>
<p>
If you've found this service useful, consider sponsoring me on
GitHub - <a href="https://github.com/sponsors/Lissy93">github.com/sponsors/Lissy93</a> 💖
</p>
</Section> </Section>
</AboutContainer> </AboutContainer>
<Footer /> <Footer />

View File

@@ -863,7 +863,7 @@ const Results = (): JSX.Element => {
} }
</Nav> </Nav>
<ProgressBar loadStatus={loadingJobs} showModal={showErrorModal} showJobDocs={showInfo} /> <ProgressBar loadStatus={loadingJobs} showModal={showErrorModal} showJobDocs={showInfo} />
{ address?.includes(window?.location?.hostname || 'web-check.xyz') && <SelfScanMsg />} { address?.includes(window?.location?.hostname || 'web-check.as93.net') && <SelfScanMsg />}
<Loader show={loadingJobs.filter((job: LoadingJob) => job.state !== 'loading').length < 5} /> <Loader show={loadingJobs.filter((job: LoadingJob) => job.state !== 'loading').length < 5} />
<FilterButtons>{ showFilters ? <> <FilterButtons>{ showFilters ? <>
<div className="one-half"> <div className="one-half">

4874
yarn.lock

File diff suppressed because it is too large Load Diff