Compare commits

..

1 Commits

Author SHA1 Message Date
Alicia Sykes
bc45b91b51 Refactored Dockerfile 2023-08-29 21:47:39 +01:00
67 changed files with 3021 additions and 4045 deletions

30
.env
View File

@@ -2,25 +2,11 @@
# Be sure to uncomment any line you populate # Be sure to uncomment any line you populate
# Everything is optional, but some features won't work without external API access # Everything is optional, but some features won't work without external API access
# API Keys for external services (backend) # GOOGLE_CLOUD_API_KEY=''
GOOGLE_CLOUD_API_KEY='' # SHODAN_API_KEY=''
TORRENT_IP_API_KEY='' # REACT_APP_SHODAN_API_KEY=''
SECURITY_TRAILS_API_KEY='' # WHO_API_KEY=''
BUILT_WITH_API_KEY='' # REACT_APP_WHO_API_KEY=''
URL_SCAN_API_KEY='' # SECURITY_TRAILS_API_KEY=''
TRANCO_USERNAME='' # BUILT_WITH_API_KEY=''
TRANCO_API_KEY='' # CI=false
CLOUDMERSIVE_API_KEY=''
# API Keys for external services (frontend)
REACT_APP_SHODAN_API_KEY=''
REACT_APP_WHO_API_KEY=''
# Configuration settings
# CHROME_PATH='/usr/bin/chromium' # The path the the Chromium executable
# PORT='3000' # Port to serve the API, when running server.js
# DISABLE_GUI='false' # Disable the GUI, and only serve the API
# API_TIMEOUT_LIMIT='10000' # The timeout limit for API requests, in milliseconds
# API_CORS_ORIGIN='*' # Enable CORS, by setting your allowed hostname(s) here
# API_ENABLE_RATE_LIMIT='true' # Enable rate limiting for the API
# REACT_APP_API_ENDPOINT='/api' # The endpoint for the API (can be local or remote)

497
.github/README.md vendored
View File

@@ -9,22 +9,6 @@
</p> </p>
---
<p align="center">
<sup>Kindly supported by:</sup><br>
<a href="https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
<img src="https://i.ibb.co/8jrrcZ0/IMG-7210.jpg" width="300" alt="Terminal Trove">
<br>
<strong>The $HOME of all things in the terminal.</strong>
</a>
<br>
<a href="https://terminaltrove.com/newsletter?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
<sub>Find your next CLI / TUI tool and more at Terminal Trove,</sub>
<br>
<sup>Get updates on new tools on our newsletter.</sup>
</a>
</p>
--- ---
#### Contents #### Contents
@@ -35,13 +19,11 @@
- [Mirror](#mirror) - [Mirror](#mirror)
- [Features](#features) - [Features](#features)
- **[Usage](#usage)** - **[Usage](#usage)**
- [Deployment](#deployment)
- [Option#1: Netlify](#deploying---option-1-netlify)
- [Option#2: Vercel](#deploying---option-2-vercel)
- [Option#3: Docker](#deploying---option-3-docker)
- [Option#4: Source](#deploying---option-4-from-source)
- [Configuration Options](#configuring)
- [Developer Setup](#developing) - [Developer Setup](#developing)
- [Deploying, Option#1: Netlify](#deploying---option-1-netlify)
- [Deploying, Option#2: Docker](#deploying---option-2-docker)
- [Deploying, Option#3: Source](#deploying---option-3-from-source)
- [Configuration Options](#configuring)
- **[Community](#community)** - **[Community](#community)**
- [Contributing](#contributing) - [Contributing](#contributing)
- [Bugs](#reporting-bugs) - [Bugs](#reporting-bugs)
@@ -74,27 +56,18 @@ A hosted version can be accessed at: **[web-check.as93.net](https://web-check.as
### Mirror ### Mirror
The source for this repo is mirrored to CodeBerg, available at: **[codeberg.org/alicia/web-check](https://codeberg.org/alicia/web-check)** The source for this repo is mirrored to CodeBerg, available at: **[codeberg.org/alicia/web-check](https://codeberg.org/alicia/web-check)**
### Status ### Motivation
Often when you're looking into a website, there's several things you always initially check.
Build & Deploys: [![Netlify Status](https://api.netlify.com/api/v1/badges/c43453c1-5333-4df7-889b-c1d2b52183c0/deploy-status)](https://app.netlify.com/sites/web-check/deploys)
[![Vercel Status](https://therealsujitk-vercel-badge.vercel.app/?app=web-check-ten)](https://vercel.com/as93/web-check/)
[![🐳 Build + Publish Docker Image](https://github.com/Lissy93/web-check/actions/workflows/docker.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/docker.yml)
[![🚀 Deploy to AWS](https://github.com/Lissy93/web-check/actions/workflows/deploy-aws.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/deploy-aws.yml)
<br />
Repo Management & Miscellaneous: [![🪞 Mirror to Codeberg](https://github.com/Lissy93/web-check/actions/workflows/mirror.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/mirror.yml)
[![💓 Inserts Contributors & Sponsors](https://github.com/Lissy93/web-check/actions/workflows/credits.yml/badge.svg)](https://github.com/Lissy93/web-check/actions/workflows/credits.yml)
None of this is hard to find with a series of basic curl commands and NPMAP plus a combination of online tools. But it's just so much easier to have everything done all at once, presented clearly and visible in one place :)
### Features ### Features
<details open> <details>
<summary><b>Click to expand / collapse section</b></summary> <summary><h4>Expand to see all features</h4></summary>
<sup>**Note** _this list needs updating, many more jobs have been added since..._</sup> <sup>**Note** _this list needs updating, many more jobs have been added since..._</sup>
The following section outlines the core features, and briefly explains why this data might be useful for you to know, as well as linking to further resources for learning more.
<details> <details>
<summary><b>IP Info</b></summary> <summary><b>IP Info</b></summary>
@@ -757,104 +730,14 @@ This may be useful to see what a given website looks like, free of the constrain
</details> </details>
Read more here: **[web-check.xyz/about](https://web-check.xyz/about)** Read more here: **[web-check.as93.net/about](https://web-check.as93.net/about)**
_Note that not all checks will work for all sites. Sometimes it's not possible to determine some information, and the demo instance has some limitations imposed by Netlify for the lambda functions._
--- ---
## Usage ## Usage
### Deployment
### Deploying - Option #1: Netlify
Click the button below, to deploy to Netlify 👇
[![Deploy to Netlify](https://img.shields.io/badge/Deploy-Netlify-%2330c8c9?style=for-the-badge&logo=netlify&labelColor=1e0e41 'Deploy Web-Check to Netlify, via 1-Click Script')](https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check)
### Deploying - Option #2: Vercel
Click the button below, to deploy to Vercel 👇
[![Deploy with Vercel](https://img.shields.io/badge/Deploy-Vercel-%23ffffff?style=for-the-badge&logo=vercel&labelColor=1e0e41)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flissy93%2Fweb-check&project-name=web-check&repository-name=web-check-fork&demo-title=Web-Check%20Demo&demo-description=Check%20out%20web-check.xyz%20to%20see%20a%20live%20demo%20of%20this%20application%20running.&demo-url=https%3A%2F%2Fweb-check.xyz&demo-image=https%3A%2F%2Fraw.githubusercontent.com%2FLissy93%2Fweb-check%2Fmaster%2F.github%2Fscreenshots%2Fweb-check-screenshot10.png)
### Deploying - Option #3: Docker
Run `docker run -p 3000:3000 lissy93/web-check`, then open [`localhost:3000`](http://localhost:3000)
<details>
<summary>Docker Options</summary>
You can get the Docker image from:
- DockerHub: [`lissy93/web-check`](https://hub.docker.com/r/lissy93/web-check)
- GHCR: [`ghcr.io/lissy93/web-check`](https://github.com/Lissy93/web-check/pkgs/container/web-check)
- Or build the image yourself by cloning the repo and running `docker build -t web-check .`
</details>
### Deploying - Option #4: From Source
Install the prerequisites listed in the [Developing](#developing) section, then run:
```bash
git clone https://github.com/Lissy93/web-check.git # Download the code from GitHub
cd web-check # Navigate into the project dir
yarn install # Install the NPM dependencies
yarn build # Build the app for production
yarn serve # Start the app (API and GUI)
```
---
### Configuring
By default, no configuration is needed.
But there are some optional environmental variables that you can set to give you access to some additional checks, or to increase rate-limits for some checks that use external APIs.
**API Keys & Credentials**:
Key | Value
---|---
`GOOGLE_CLOUD_API_KEY` | A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
`REACT_APP_SHODAN_API_KEY` | A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
`REACT_APP_WHO_API_KEY` | A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
<details>
<summary><small>Full / Upcoming Vals</small></summary>
- `GOOGLE_CLOUD_API_KEY` - A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
- `REACT_APP_SHODAN_API_KEY` - A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
- `REACT_APP_WHO_API_KEY` - A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
- `SECURITY_TRAILS_API_KEY` - A Security Trails API key ([get here](https://securitytrails.com/corp/api)). This will show org info associated with the IP
- `CLOUDMERSIVE_API_KEY` - API key for Cloudmersive ([get here](https://account.cloudmersive.com/)). This will show known threats associated with the IP
- `TRANCO_USERNAME` - A Tranco email ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `TRANCO_API_KEY` - A Tranco API key ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `URL_SCAN_API_KEY` - A URLScan API key ([get here](https://urlscan.io/)). This will fetch miscalanious info about a site
- `BUILT_WITH_API_KEY` - A BuiltWith API key ([get here](https://api.builtwith.com/)). This will show the main features of a site
- `TORRENT_IP_API_KEY` - A torrent API key ([get here](https://iknowwhatyoudownload.com/en/api/)). This will show torrents downloaded by an IP
</details>
**Configuration Settings**:
Key | Value
---|---
`PORT` | Port to serve the API, when running server.js (e.g. `3000`)
`API_ENABLE_RATE_LIMIT` | Enable rate-limiting for the /api endpoints (e.g. `true`)
`API_TIMEOUT_LIMIT` | The timeout limit for API requests, in milliseconds (e.g. `10000`)
`API_CORS_ORIGIN` | Enable CORS, by setting your allowed hostname(s) here (e.g. `example.com`)
`CHROME_PATH` | The path the Chromium executable (e.g. `/usr/bin/chromium`)
`DISABLE_GUI` | Disable the GUI, and only serve the API (e.g. `false`)
`REACT_APP_API_ENDPOINT` | The endpoint for the API, either local or remote (e.g. `/api`)
All values are optional.
You can add these as environmental variables. Either put them directly into an `.env` file in the projects root, or via the Netlify / Vercel UI, or by passing to the Docker container with the --env flag, or using your own environmental variable management system
Note that keys that are prefixed with `REACT_APP_` are used client-side, and as such they must be scoped correctly with minimum privileges, since may be made visible when intercepting browser <-> server network requests
---
### Developing ### Developing
1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git` 1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git`
@@ -865,6 +748,64 @@ Note that keys that are prefixed with `REACT_APP_` are used client-side, and as
You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/). You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/).
Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present. Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present.
### Deploying - Option #1: Netlify
Click the button below, to deploy to Netlify 👇
[![Deploy to Netlify](https://img.shields.io/badge/Deploy-Netlify-%2330c8c9?style=for-the-badge&logo=netlify&labelColor=1e0e41 'Deploy Web-Check to Netlify, via 1-Click Script')](https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check)
### Deploying - Option #2: Docker
Run `docker run -p 8888:3000 lissy93/web-check`, then open `http://localhost:3000`
You can get the Docker image from:
- DockerHub: [`lissy93/web-check`](https://hub.docker.com/r/lissy93/web-check)
- GHCR: [`ghcr.io/lissy93/web-check`](https://github.com/Lissy93/web-check/pkgs/container/web-check)
- Or build the image yourself by cloning the repo and running `docker build -t web-check .`
### Deploying - Option #3: From Source
Follow the instructions in the [Developing](#developing) section above, then run `yarn build` && `yarn serve` to build and serve the application.
The following commands will work:
```bash
git clone https://github.com/Lissy93/web-check.git # Grab the code
cd web-check # Move into the project directory
yarn install # Install dependencies
yarn build # Build the app for production
yarn serve # Start the app (API and GUI)
```
### Configuring
By default, no configuration is needed.
But there are some optional environmental variables that you can set to give you access to some additional checks, or to increase rate-limits for some checks that use external APIs.
Note that keys that are prefixed with `REACT_APP_` are used client-side, and as such they must be scoped correctly with minimum privileges.
**API Keys & Credentials**:
- `GOOGLE_CLOUD_API_KEY` - A Google API key ([get here](https://cloud.google.com/api-gateway/docs/authenticate-api-keys)). This can be used to return quality metrics for a site
- `REACT_APP_SHODAN_API_KEY` - A Shodan API key ([get here](https://account.shodan.io/)). This will show associated host names for a given domain
- `REACT_APP_WHO_API_KEY` - A WhoAPI key ([get here](https://whoapi.com/)). This will show more comprehensive WhoIs records than the default job
- `SECURITY_TRAILS_API_KEY` - A Security Trails API key ([get here](https://securitytrails.com/corp/api)). This will show org info associated with the IP
- `CLOUDMERSIVE_API_KEY` - API key for Cloudmersive ([get here](https://account.cloudmersive.com/)). This will show known threats associated with the IP
- `TRANCO_USERNAME` - A Tranco email ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `TRANCO_API_KEY` - A Tranco API key ([get here](https://tranco-list.eu/)). This will show the rank of a site, based on traffic
- `URL_SCAN_API_KEY` - A URLScan API key ([get here](https://urlscan.io/)). This will fetch miscalanious info about a site
- `BUILT_WITH_API_KEY` - A BuiltWith API key ([get here](https://api.builtwith.com/)). This will show the main features of a site
- `TORRENT_IP_API_KEY` - A torrent API key ([get here](https://iknowwhatyoudownload.com/en/api/)). This will show torrents downloaded by an IP
**Configuration Settings**:
- `CHROME_PATH` (e.g. `/usr/bin/chromium`) - The path the the Chromium executable
- `PORT` (e.g. `3000`) - Port to serve the API, when running server.js
- `DISABLE_GUI` (e.g. `false`) - Disable the GUI, and only serve the API
- `API_TIMEOUT_LIMIT` (e.g. `10000`) - The timeout limit for API requests, in milliseconds
- `REACT_APP_API_ENDPOINT` (e.g. `/api`) - The endpoint for the API (can be local or remote)
The above can be added into an `.env` file in the projects root, or via the Netlify UI, or by passing directly to the Docker container with the --env flag.
All variables are optional.
--- ---
@@ -890,9 +831,8 @@ For bugs, please outline the steps needed to reproduce, and include relevant inf
### Supporting ### Supporting
The app will remain 100% free and open source. The app will remain 100% free and open source.
But due to the amount of traffic that the hosted instance gets, the lambda function usage is costing about $25/month. But due to the amount of traffic that the hosted instance gets, the lambda function usage is costing about $25/month. Any help with covering the costs via GitHub Sponsorship would be much appreciated. It's thanks to the support of the community that this project is able to be freely available for everyone :)
Any help with covering the costs via GitHub Sponsorship would be much appreciated.
It's thanks to the support of the community that this project is able to be freely available for everyone :)
[![Sponsor Lissy93 on GitHub](https://img.shields.io/badge/Sponsor_on_GitHub-Lissy93-%23ff4dda?style=for-the-badge&logo=githubsponsors&logoColor=ff4dda)](https://github.com/sponsors/Lissy93) [![Sponsor Lissy93 on GitHub](https://img.shields.io/badge/Sponsor_on_GitHub-Lissy93-%23ff4dda?style=for-the-badge&logo=githubsponsors&logoColor=ff4dda)](https://github.com/sponsors/Lissy93)
@@ -911,13 +851,6 @@ Credit to the following users for contributing to Web-Check
<sub><b>Alicia Sykes</b></sub> <sub><b>Alicia Sykes</b></sub>
</a> </a>
</td> </td>
<td align="center">
<a href="https://github.com/liss-bot">
<img src="https://avatars.githubusercontent.com/u/87835202?v=4" width="80;" alt="liss-bot"/>
<br />
<sub><b>Alicia Bot</b></sub>
</a>
</td>
<td align="center"> <td align="center">
<a href="https://github.com/muni106"> <a href="https://github.com/muni106">
<img src="https://avatars.githubusercontent.com/u/65845442?v=4" width="80;" alt="muni106"/> <img src="https://avatars.githubusercontent.com/u/65845442?v=4" width="80;" alt="muni106"/>
@@ -926,89 +859,10 @@ Credit to the following users for contributing to Web-Check
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/ChrisCarini"> <a href="https://github.com/gso-trifork-security">
<img src="https://avatars.githubusercontent.com/u/6374067?v=4" width="80;" alt="ChrisCarini"/> <img src="https://avatars.githubusercontent.com/u/69247026?v=4" width="80;" alt="gso-trifork-security"/>
<br /> <br />
<sub><b>Chris Carini</b></sub> <sub><b>Gustav Soelberg</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/brianteeman">
<img src="https://avatars.githubusercontent.com/u/1296369?v=4" width="80;" alt="brianteeman"/>
<br />
<sub><b>Brian Teeman</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/GreyXor">
<img src="https://avatars.githubusercontent.com/u/79602273?v=4" width="80;" alt="GreyXor"/>
<br />
<sub><b>GreyXor</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/jinnabaalu">
<img src="https://avatars.githubusercontent.com/u/11784253?v=4" width="80;" alt="jinnabaalu"/>
<br />
<sub><b>Jinna Baalu</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/bolens">
<img src="https://avatars.githubusercontent.com/u/1218380?v=4" width="80;" alt="bolens"/>
<br />
<sub><b>Michael Bolens</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/robinson">
<img src="https://avatars.githubusercontent.com/u/237874?v=4" width="80;" alt="robinson"/>
<br />
<sub><b>Lth</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/abhishekMuge">
<img src="https://avatars.githubusercontent.com/u/49590582?v=4" width="80;" alt="abhishekMuge"/>
<br />
<sub><b>Abhishek Muge</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/UlisesGascon">
<img src="https://avatars.githubusercontent.com/u/5110813?v=4" width="80;" alt="UlisesGascon"/>
<br />
<sub><b>Ulises Gascón</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/PhiRequiem">
<img src="https://avatars.githubusercontent.com/u/1323576?v=4" width="80;" alt="PhiRequiem"/>
<br />
<sub><b>PhiRequiem</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/Myzel394">
<img src="https://avatars.githubusercontent.com/u/50424412?v=4" width="80;" alt="Myzel394"/>
<br />
<sub><b>Myzel394</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/murrple-1">
<img src="https://avatars.githubusercontent.com/u/5559656?v=4" width="80;" alt="murrple-1"/>
<br />
<sub><b>Murray Christopherson</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/t3chn0m4g3">
<img src="https://avatars.githubusercontent.com/u/4318452?v=4" width="80;" alt="t3chn0m4g3"/>
<br />
<sub><b>Marco Ochse</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
@@ -1019,32 +873,17 @@ Credit to the following users for contributing to Web-Check
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/eltociear"> <a href="https://github.com/UlisesGascon">
<img src="https://avatars.githubusercontent.com/u/22633385?v=4" width="80;" alt="eltociear"/> <img src="https://avatars.githubusercontent.com/u/5110813?v=4" width="80;" alt="UlisesGascon"/>
<br /> <br />
<sub><b>Ikko Eltociear Ashimine</b></sub> <sub><b>Ulises Gascón</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/Gertje823"> <a href="https://github.com/abhishekMuge">
<img src="https://avatars.githubusercontent.com/u/36937387?v=4" width="80;" alt="Gertje823"/> <img src="https://avatars.githubusercontent.com/u/49590582?v=4" width="80;" alt="abhishekMuge"/>
<br /> <br />
<sub><b>Gertje823</b></sub> <sub><b>Abhishek Muge</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/epreston">
<img src="https://avatars.githubusercontent.com/u/347224?v=4" width="80;" alt="epreston"/>
<br />
<sub><b>Ed Preston</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/0xflotus">
<img src="https://avatars.githubusercontent.com/u/26602940?v=4" width="80;" alt="0xflotus"/>
<br />
<sub><b>0xflotus</b></sub>
</a> </a>
</td></tr> </td></tr>
</table> </table>
@@ -1057,204 +896,111 @@ Huge thanks to these wonderful people, who sponsor me on GitHub, their support h
<!-- readme: sponsors -start --> <!-- readme: sponsors -start -->
<table> <table>
<tr> <tr>
<td align="center">
<a href="https://github.com/github">
<img src="https://avatars.githubusercontent.com/u/9919?v=4" width="80;" alt="github"/>
<br />
<sub><b>GitHub</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/koconder">
<img src="https://avatars.githubusercontent.com/u/25068?u=582657b23622aaa3dfe68bd028a780f272f456fa&v=4" width="80;" alt="koconder"/>
<br />
<sub><b>Vincent Koc</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Admonstrator">
<img src="https://avatars.githubusercontent.com/u/69824?u=1e226d7a36cdd661c3e4cd486fea140d045b7d57&v=4" width="80;" alt="Admonstrator"/>
<br />
<sub><b>Aaron Viehl</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/tbjers">
<img src="https://avatars.githubusercontent.com/u/1117052?v=4" width="80;" alt="tbjers"/>
<br />
<sub><b>Torgny Bjers</b></sub>
</a>
</td>
<td align="center"> <td align="center">
<a href="https://github.com/emlazzarin"> <a href="https://github.com/emlazzarin">
<img src="https://avatars.githubusercontent.com/u/1141361?u=714e3487a3f2e0df721b01a0133945f075d3ff68&v=4" width="80;" alt="emlazzarin"/> <img src="https://avatars.githubusercontent.com/u/1141361?v=4" width="80;" alt="emlazzarin"/>
<br /> <br />
<sub><b>Eddy Lazzarin</b></sub> <sub><b>Eddy Lazzarin</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/AnandChowdhary"> <a href="https://github.com/AnandChowdhary">
<img src="https://avatars.githubusercontent.com/u/2841780?u=747e554b3a7f12eb20b7910e1c87d817844f714f&v=4" width="80;" alt="AnandChowdhary"/> <img src="https://avatars.githubusercontent.com/u/2841780?v=4" width="80;" alt="AnandChowdhary"/>
<br /> <br />
<sub><b>Anand Chowdhary</b></sub> <sub><b>Anand Chowdhary</b></sub>
</a> </a>
</td></tr> </td>
<tr>
<td align="center"> <td align="center">
<a href="https://github.com/shrippen"> <a href="https://github.com/shrippen">
<img src="https://avatars.githubusercontent.com/u/2873570?v=4" width="80;" alt="shrippen"/> <img src="https://avatars.githubusercontent.com/u/2873570?v=4" width="80;" alt="shrippen"/>
<br /> <br />
<sub><b>Shrippen</b></sub> <sub><b>shrippen</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/davidpaulyoung">
<img src="https://avatars.githubusercontent.com/u/3418369?v=4" width="80;" alt="davidpaulyoung"/>
<br />
<sub><b>David Young</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/k-rol">
<img src="https://avatars.githubusercontent.com/u/4050412?v=4" width="80;" alt="k-rol"/>
<br />
<sub><b>Carol Ouellet</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/bile0026"> <a href="https://github.com/bile0026">
<img src="https://avatars.githubusercontent.com/u/5022496?u=aec96ad173c0ea9baaba93807efa8a848af6595c&v=4" width="80;" alt="bile0026"/> <img src="https://avatars.githubusercontent.com/u/5022496?v=4" width="80;" alt="bile0026"/>
<br /> <br />
<sub><b>Zach Biles</b></sub> <sub><b>Zach Biles</b></sub>
</a> </a>
</td> </td></tr>
<tr>
<td align="center"> <td align="center">
<a href="https://github.com/UlisesGascon"> <a href="https://github.com/UlisesGascon">
<img src="https://avatars.githubusercontent.com/u/5110813?u=3c41facd8aa26154b9451de237c34b0f78d672a5&v=4" width="80;" alt="UlisesGascon"/> <img src="https://avatars.githubusercontent.com/u/5110813?v=4" width="80;" alt="UlisesGascon"/>
<br /> <br />
<sub><b>Ulises Gascón</b></sub> <sub><b>Ulises Gascón</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/digitalarche"> <a href="https://github.com/digitalarche">
<img src="https://avatars.githubusercontent.com/u/6546135?u=564756d7f44ab2206819eb3148f6d822673f5066&v=4" width="80;" alt="digitalarche"/> <img src="https://avatars.githubusercontent.com/u/6546135?v=4" width="80;" alt="digitalarche"/>
<br /> <br />
<sub><b>Digital Archeology</b></sub> <sub><b>Digital Archeology</b></sub>
</a> </a>
</td> </td>
<td align="center">
<a href="https://github.com/InDieTasten">
<img src="https://avatars.githubusercontent.com/u/7047377?u=8d8f8017628b38bc46dcbf3620e194b01d3fb2d1&v=4" width="80;" alt="InDieTasten"/>
<br />
<sub><b>InDieTasten</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/araguaci">
<img src="https://avatars.githubusercontent.com/u/7318668?v=4" width="80;" alt="araguaci"/>
<br />
<sub><b>Araguaci</b></sub>
</a>
</td></tr>
<tr>
<td align="center"> <td align="center">
<a href="https://github.com/bmcgonag"> <a href="https://github.com/bmcgonag">
<img src="https://avatars.githubusercontent.com/u/7346620?u=2a0f9284f3e12ac1cc15288c254d1ec68a5081e8&v=4" width="80;" alt="bmcgonag"/> <img src="https://avatars.githubusercontent.com/u/7346620?v=4" width="80;" alt="bmcgonag"/>
<br /> <br />
<sub><b>Brian McGonagill</b></sub> <sub><b>Brian McGonagill</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/vlad-timofeev"> <a href="https://github.com/vlad-timofeev">
<img src="https://avatars.githubusercontent.com/u/11474041?u=eee43705b54d2ec9f51fc4fcce5ad18dd17c87e4&v=4" width="80;" alt="vlad-timofeev"/> <img src="https://avatars.githubusercontent.com/u/11474041?v=4" width="80;" alt="vlad-timofeev"/>
<br /> <br />
<sub><b>Vlad Timofeev</b></sub> <sub><b>Vlad Timofeev</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/helixzz"> <a href="https://github.com/iJasonWade">
<img src="https://avatars.githubusercontent.com/u/12218889?u=d06d0c103dfbdb99450623064f7da3c5a3675fb6&v=4" width="80;" alt="helixzz"/> <img src="https://avatars.githubusercontent.com/u/12824479?v=4" width="80;" alt="iJasonWade"/>
<br /> <br />
<sub><b>HeliXZz</b></sub> <sub><b>Jason Ash</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/patvdv"> <a href="https://github.com/DRXAquosus">
<img src="https://avatars.githubusercontent.com/u/12430107?u=e8911c2fb91af4d30432f76da8c40927b2830bd7&v=4" width="80;" alt="patvdv"/> <img src="https://avatars.githubusercontent.com/u/45409262?v=4" width="80;" alt="DRXAquosus"/>
<br /> <br />
<sub><b>Patrick Van Der Veken</b></sub> <sub><b>DRXAquosus</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/plgonzalezrx8">
<img src="https://avatars.githubusercontent.com/u/19900049?u=48a58d2da520a9d712184c6e6e99927ff3cbf179&v=4" width="80;" alt="plgonzalezrx8"/>
<br />
<sub><b>Pedro Gonzalez</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/mryesiller">
<img src="https://avatars.githubusercontent.com/u/24632172?u=0d20f2d615158f87cd60a3398d3efb026c32f291&v=4" width="80;" alt="mryesiller"/>
<br />
<sub><b>Göksel Yeşiller</b></sub>
</a> </a>
</td></tr> </td></tr>
<tr> <tr>
<td align="center">
<a href="https://github.com/allesauseinerhand">
<img src="https://avatars.githubusercontent.com/u/32039836?v=4" width="80;" alt="allesauseinerhand"/>
<br />
<sub><b>Allesauseinerhand</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/forwardemail">
<img src="https://avatars.githubusercontent.com/u/32481436?v=4" width="80;" alt="forwardemail"/>
<br />
<sub><b>Forward Email - Open-source & Privacy-focused Email Service (2023)</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/lamtrinhdev">
<img src="https://avatars.githubusercontent.com/u/49742151?u=c5eaca5aa6841a80605cf4f7d0e861a9e6339ef3&v=4" width="80;" alt="lamtrinhdev"/>
<br />
<sub><b>LamTrinh.Dev</b></sub>
</a>
</td>
<td align="center"> <td align="center">
<a href="https://github.com/Bastii717"> <a href="https://github.com/Bastii717">
<img src="https://avatars.githubusercontent.com/u/53431819?u=604977bed6ad6875ada890d0d3765a4cacc2fa14&v=4" width="80;" alt="Bastii717"/> <img src="https://avatars.githubusercontent.com/u/53431819?v=4" width="80;" alt="Bastii717"/>
<br /> <br />
<sub><b>Bastii717</b></sub> <sub><b>Bastii717</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/getumbrel"> <a href="https://github.com/ratty222">
<img src="https://avatars.githubusercontent.com/u/59408891?v=4" width="80;" alt="getumbrel"/> <img src="https://avatars.githubusercontent.com/u/92832598?v=4" width="80;" alt="ratty222"/>
<br /> <br />
<sub><b>Umbrel</b></sub> <sub><b>Brent</b></sub>
</a> </a>
</td> </td>
<td align="center"> <td align="center">
<a href="https://github.com/M2TD"> <a href="https://github.com/jtfinley72">
<img src="https://avatars.githubusercontent.com/u/85460457?v=4" width="80;" alt="M2TD"/> <img src="https://avatars.githubusercontent.com/u/96497997?v=4" width="80;" alt="jtfinley72"/>
<br /> <br />
<sub><b>M2TD</b></sub> <sub><b>jtfinley72</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/frankdez93">
<img src="https://avatars.githubusercontent.com/u/87549420?v=4" width="80;" alt="frankdez93"/>
<br />
<sub><b>Frankdez93</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/terminaltrove">
<img src="https://avatars.githubusercontent.com/u/121595180?v=4" width="80;" alt="terminaltrove"/>
<br />
<sub><b>Terminal Trove</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/NixyJuppie">
<img src="https://avatars.githubusercontent.com/u/138570196?u=b102c222487905728b858704962d32759df29ebe&v=4" width="80;" alt="NixyJuppie"/>
<br />
<sub><b>Nixy</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/nrvo">
<img src="https://avatars.githubusercontent.com/u/151435968?u=e1dcb307fd0efdc45cddbe9490a7b956e4da6835&v=4" width="80;" alt="nrvo"/>
<br />
<sub><b>Nrvo</b></sub>
</a> </a>
</td></tr> </td></tr>
</table> </table>
@@ -1292,11 +1038,8 @@ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
``` ```
[![View Dependency Licenses & SBOM on FOSSA](https://app.fossa.com/api/projects/git%2Bgithub.com%2FLissy93%2Fweb-check.svg?type=large&issueType=license)](https://app.fossa.com/projects/git%2Bgithub.com%2FLissy93%2Fweb-check?ref=badge_large&issueType=license)
</details> </details>
<!-- License + Copyright --> <!-- License + Copyright -->
<p align="center"> <p align="center">
<i>© <a href="https://aliciasykes.com">Alicia Sykes</a> 2023</i><br> <i>© <a href="https://aliciasykes.com">Alicia Sykes</a> 2023</i><br>
@@ -1305,7 +1048,7 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
<sup>Thanks for visiting :)</sup> <sup>Thanks for visiting :)</sup>
</p> </p>
<!-- Dinosaurs are Awesome --> <!-- Dinosaur -->
<!-- <!--
. - ~ ~ ~ - . . - ~ ~ ~ - .
.. _ .-~ ~-. .. _ .-~ ~-.

View File

@@ -1,128 +0,0 @@
name: 🚀 Deploy to AWS
on:
workflow_dispatch:
push:
branches:
- master
tags:
- '*'
paths:
- api/**
- serverless.yml
- package.json
- .github/workflows/deploy-aws.yml
jobs:
deploy-api:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v1
with:
node-version: 16
- name: Cache node_modules
uses: actions/cache@v2
with:
path: node_modules
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Create GitHub deployment for API
uses: chrnorm/deployment-action@releases/v1
id: deployment_api
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
environment: AWS (Backend API)
ref: ${{ github.ref }}
- name: Install Serverless CLI and dependencies
run: |
npm i -g serverless
yarn
- name: Deploy to AWS
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
run: serverless deploy
- name: Update GitHub deployment status (API)
if: always()
uses: chrnorm/deployment-status@releases/v1
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
state: "${{ job.status }}"
deployment_id: ${{ steps.deployment_api.outputs.deployment_id }}
ref: ${{ github.ref }}
deploy-frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v1
with:
node-version: 16
- name: Cache node_modules
uses: actions/cache@v2
with:
path: node_modules
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Create GitHub deployment for Frontend
uses: chrnorm/deployment-action@releases/v1
id: deployment_frontend
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
environment: AWS (Frontend Web UI)
ref: ${{ github.ref }}
- name: Install dependencies and build
run: |
yarn install
yarn build
- name: Setup AWS
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Upload to S3
env:
AWS_S3_BUCKET: 'web-check-frontend'
run: aws s3 sync ./build/ s3://$AWS_S3_BUCKET/ --delete
- name: Invalidate CloudFront cache
uses: chetan/invalidate-cloudfront-action@v2.4
env:
DISTRIBUTION: E30XKAM2TG9FD8
PATHS: '/*'
AWS_REGION: 'us-east-1'
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Update GitHub deployment status (Frontend)
if: always()
uses: chrnorm/deployment-status@releases/v1
with:
token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
state: "${{ job.status }}"
deployment_id: ${{ steps.deployment_frontend.outputs.deployment_id }}
ref: ${{ github.ref }}

View File

@@ -67,7 +67,7 @@ jobs:
context: . context: .
file: ./Dockerfile file: ./Dockerfile
push: true push: true
platforms: linux/amd64,linux/arm64/v8 platforms: linux/amd64
tags: | tags: |
${{ env.GHCR_TAG }} ${{ env.GHCR_TAG }}
${{ env.DOCKERHUB_TAG }} ${{ env.DOCKERHUB_TAG }}

69
.gitignore vendored
View File

@@ -1,60 +1,27 @@
# ------------------------
# ENVIRONMENT SETTINGS # keys
# ------------------------
.env .env
# ------------------------ # dependencies
# PRODUCTION /node_modules
# ------------------------ /.pnp
/build/ .pnp.js
# ------------------------ # logs
# DEPLOYMENT
# ------------------------
.vercel/
.netlify/
.webpack/
.serverless/
# ------------------------
# DEPENDENCIES
# ------------------------
node_modules/
.yarn/cache/
.yarn/unplugged/
.yarn/build-state.yml
.yarn/install-state.gz
.pnpm/
.pnp.*
# ------------------------
# LOGS
# ------------------------
logs/
*.log
npm-debug.log* npm-debug.log*
yarn-debug.log* yarn-debug.log*
yarn-error.log* yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
# ------------------------ # testing
# TESTING /coverage
# ------------------------
coverage/
.nyc_output/
# ------------------------ # production
# OS SPECIFIC /build
# ------------------------
# Random AWS and Netlify crap
.netlify
.serverless
.webpack
# OS generated files
.DS_Store .DS_Store
Thumbs.db
# ------------------------
# EDITORS
# ------------------------
.idea/
.vscode/
*.swp
*.swo

View File

@@ -1,62 +1,22 @@
# Specify the Node.js version to use FROM node:16-buster-slim
ARG NODE_VERSION=16
# Specify the Debian version to use, the default is "bullseye" RUN apt-get update && \
ARG DEBIAN_VERSION=bullseye apt-get install -y chromium traceroute && \
chmod 755 /usr/bin/chromium && \
# Use Node.js Docker image as the base image, with specific Node and Debian versions
FROM node:${NODE_VERSION}-${DEBIAN_VERSION} AS build
# Set the container's default shell to Bash and enable some options
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
# Install Chromium browser and Download and verify Google Chromes signing key
RUN apt-get update -qq --fix-missing && \
apt-get -qqy install --allow-unauthenticated gnupg wget && \
wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google.list && \
apt-get update -qq && \
apt-get -qqy --no-install-recommends install chromium traceroute python make g++ && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
# Run the Chromium browser's version command and redirect its output to the /etc/chromium-version file
RUN /usr/bin/chromium --no-sandbox --version > /etc/chromium-version
# Set the working directory to /app
WORKDIR /app WORKDIR /app
# Copy package.json and yarn.lock to the working directory
COPY package.json yarn.lock ./ COPY package.json yarn.lock ./
# Run yarn install to install dependencies and clear yarn cache RUN yarn install
RUN apt-get update && \
yarn install --production --frozen-lockfile --network-timeout 100000 && \
rm -rf /app/node_modules/.cache
# Copy all files to working directory
COPY . . COPY . .
# Run yarn build to build the application RUN yarn build
RUN yarn build --production
# Final stage
FROM node:${NODE_VERSION}-${DEBIAN_VERSION} AS final
WORKDIR /app
COPY package.json yarn.lock ./
COPY --from=build /app .
RUN apt-get update && \
apt-get install -y --no-install-recommends chromium traceroute && \
chmod 755 /usr/bin/chromium && \
rm -rf /var/lib/apt/lists/* /app/node_modules/.cache
# Exposed container port, the default is 3000, which can be modified through the environment variable PORT
EXPOSE ${PORT:-3000} EXPOSE ${PORT:-3000}
# Set the environment variable CHROME_PATH to specify the path to the Chromium binaries
ENV CHROME_PATH='/usr/bin/chromium' ENV CHROME_PATH='/usr/bin/chromium'
# Define the command executed when the container starts and start the server.js of the Node.js application
CMD ["yarn", "serve"] CMD ["yarn", "serve"]

View File

@@ -2,84 +2,8 @@ const normalizeUrl = (url) => {
return url.startsWith('http') ? url : `https://${url}`; return url.startsWith('http') ? url : `https://${url}`;
}; };
// If present, set a shorter timeout for API requests
const TIMEOUT = process.env.API_TIMEOUT_LIMIT ? parseInt(process.env.API_TIMEOUT_LIMIT, 10) : 60000;
// If present, set CORS allowed origins for responses
const ALLOWED_ORIGINS = process.env.API_CORS_ORIGIN || '*';
// Set the platform currently being used
let PLATFORM = 'NETLIFY';
if (process.env.PLATFORM) { PLATFORM = process.env.PLATFORM.toUpperCase(); }
else if (process.env.VERCEL) { PLATFORM = 'VERCEL'; }
else if (process.env.WC_SERVER) { PLATFORM = 'NODE'; }
// Define the headers to be returned with each response
const headers = {
'Access-Control-Allow-Origin': ALLOWED_ORIGINS,
'Access-Control-Allow-Credentials': true,
'Content-Type': 'application/json;charset=UTF-8',
};
const timeoutErrorMsg = 'You can re-trigger this request, by clicking "Retry"\n'
+ 'If you\'re running your own instance of Web Check, then you can '
+ 'resolve this issue, by increasing the timeout limit in the '
+ '`API_TIMEOUT_LIMIT` environmental variable to a higher value (in milliseconds), '
+ 'or if you\'re hosting on Vercel increase the maxDuration in vercel.json.\n\n'
+ `The public instance currently has a lower timeout of ${TIMEOUT}ms `
+ 'in order to keep running costs affordable, so that Web Check can '
+ 'remain freely available for everyone.';
// A middleware function used by all API routes on all platforms
const commonMiddleware = (handler) => { const commonMiddleware = (handler) => {
return async (event, context, callback) => {
// Create a timeout promise, to throw an error if a request takes too long
const createTimeoutPromise = (timeoutMs) => {
return new Promise((_, reject) => {
setTimeout(() => {
reject(new Error(`Request timed-out after ${timeoutMs} ms`));
}, timeoutMs);
});
};
// Vercel
const vercelHandler = async (request, response) => {
const queryParams = request.query || {};
const rawUrl = queryParams.url;
if (!rawUrl) {
return response.status(500).json({ error: 'No URL specified' });
}
const url = normalizeUrl(rawUrl);
try {
// Race the handler against the timeout
const handlerResponse = await Promise.race([
handler(url, request),
createTimeoutPromise(TIMEOUT)
]);
if (handlerResponse.body && handlerResponse.statusCode) {
response.status(handlerResponse.statusCode).json(handlerResponse.body);
} else {
response.status(200).json(
typeof handlerResponse === 'object' ? handlerResponse : JSON.parse(handlerResponse)
);
}
} catch (error) {
let errorCode = 500;
if (error.message.includes('timed-out') || response.statusCode === 504) {
errorCode = 408;
error.message = `${error.message}\n\n${timeoutErrorMsg}`;
}
response.status(errorCode).json({ error: error.message });
}
};
// Netlify
const netlifyHandler = async (event, context, callback) => {
const queryParams = event.queryStringParameters || event.query || {}; const queryParams = event.queryStringParameters || event.query || {};
const rawUrl = queryParams.url; const rawUrl = queryParams.url;
@@ -87,41 +11,28 @@ const commonMiddleware = (handler) => {
callback(null, { callback(null, {
statusCode: 500, statusCode: 500,
body: JSON.stringify({ error: 'No URL specified' }), body: JSON.stringify({ error: 'No URL specified' }),
headers,
}); });
return;
} }
const url = normalizeUrl(rawUrl); const url = normalizeUrl(rawUrl);
try { try {
// Race the handler against the timeout const response = await handler(url, event, context);
const handlerResponse = await Promise.race([ if (response.body && response.statusCode) {
handler(url, event, context), callback(null, response);
createTimeoutPromise(TIMEOUT)
]);
if (handlerResponse.body && handlerResponse.statusCode) {
callback(null, handlerResponse);
} else { } else {
callback(null, { callback(null, {
statusCode: 200, statusCode: 200,
body: typeof handlerResponse === 'object' ? JSON.stringify(handlerResponse) : handlerResponse, body: typeof response === 'object' ? JSON.stringify(response) : response,
headers,
}); });
} }
} catch (error) { } catch (error) {
callback(null, { callback(null, {
statusCode: 500, statusCode: 500,
body: JSON.stringify({ error: error.message }), body: JSON.stringify({ error: error.message }),
headers,
}); });
} }
}; };
// The format of the handlers varies between platforms
const nativeMode = (['VERCEL', 'NODE'].includes(PLATFORM));
return nativeMode ? vercelHandler : netlifyHandler;
}; };
module.exports = commonMiddleware; module.exports = commonMiddleware;

View File

@@ -80,5 +80,4 @@ const getWaybackData = async (url) => {
} }
}; };
module.exports = middleware(getWaybackData); exports.handler = middleware(getWaybackData);
module.exports.handler = middleware(getWaybackData);

View File

@@ -94,12 +94,12 @@ const checkDomainAgainstDnsServers = async (domain) => {
return results; return results;
}; };
const handler = async (url) => { exports.handler = middleware(async (url) => {
const domain = new URL(url).hostname; const domain = new URL(url).hostname;
const results = await checkDomainAgainstDnsServers(domain); const results = await checkDomainAgainstDnsServers(domain);
return { blocklists: results }; return {
statusCode: 200,
body: JSON.stringify({ blocklists: results })
}; };
});
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -48,5 +48,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -54,5 +54,4 @@ const handler = async (url) => {
return { headerCookies, clientCookies }; return { headerCookies, clientCookies };
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const dns = require('dns'); const dns = require('dns');
const dnsPromises = dns.promises; const dnsPromises = dns.promises;
const axios = require('axios'); const axios = require('axios');
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const handler = async (url) => { const handler = async (url) => {
try { try {
@@ -41,5 +41,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -51,5 +51,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); // Make sure this path is correct const commonMiddleware = require('./_common/middleware'); // Make sure this path is correct
const handler = async (domain) => { const fetchDNSRecords = async (domain, event, context) => {
const dnsTypes = ['DNSKEY', 'DS', 'RRSIG']; const dnsTypes = ['DNSKEY', 'DS', 'RRSIG'];
const records = {}; const records = {};
@@ -25,11 +25,7 @@ const handler = async (domain) => {
}); });
res.on('end', () => { res.on('end', () => {
try {
resolve(JSON.parse(data)); resolve(JSON.parse(data));
} catch (error) {
reject(new Error('Invalid JSON response'));
}
}); });
res.on('error', error => { res.on('error', error => {
@@ -53,5 +49,4 @@ const handler = async (domain) => {
return records; return records;
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(fetchDNSRecords);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const builtWithHandler = async (url) => {
const apiKey = process.env.BUILT_WITH_API_KEY; const apiKey = process.env.BUILT_WITH_API_KEY;
if (!url) { if (!url) {
@@ -45,5 +45,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(builtWithHandler);
module.exports.handler = middleware(handler);

View File

@@ -102,5 +102,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -18,6 +18,4 @@ const handler = async (url) => {
return await lookupAsync(address); return await lookupAsync(address);
}; };
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -15,5 +15,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,15 +1,18 @@
const https = require('https'); const https = require('https');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url, event, context) => { exports.handler = middleware(async (url, event, context) => {
const errorResponse = (message, statusCode = 500) => { const errorResponse = (message, statusCode = 500) => {
return { return {
statusCode: statusCode, statusCode: statusCode,
body: JSON.stringify({ error: message }), body: JSON.stringify({ error: message }),
}; };
}; };
const hstsIncompatible = (message, compatible = false, hstsHeader = null ) => { const hstsIncompatible = (message, statusCode = 200) => {
return { message, compatible, hstsHeader }; return {
statusCode: statusCode,
body: JSON.stringify({ message, compatible: false }),
};
}; };
@@ -32,7 +35,14 @@ const handler = async (url, event, context) => {
} else if (!preload) { } else if (!preload) {
resolve(hstsIncompatible(`HSTS header does not contain the preload directive.`)); resolve(hstsIncompatible(`HSTS header does not contain the preload directive.`));
} else { } else {
resolve(hstsIncompatible(`Site is compatible with the HSTS preload list!`, true, hstsHeader)); resolve({
statusCode: 200,
body: JSON.stringify({
message: "Site is compatible with the HSTS preload list!",
compatible: true,
hstsHeader: hstsHeader,
}),
});
} }
} }
}); });
@@ -43,8 +53,5 @@ const handler = async (url, event, context) => {
req.end(); req.end();
}); });
}; });
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -8,7 +8,7 @@ const handler = async (url) => {
const response = await axios.get(fullUrl); const response = await axios.get(fullUrl);
const headers = response.headers; const headers = response.headers;
return { return {
strictTransportPolicy: headers['strict-transport-security'] ? true : false, strictTransportPolicy: headers['strict-transport-policy'] ? true : false,
xFrameOptions: headers['x-frame-options'] ? true : false, xFrameOptions: headers['x-frame-options'] ? true : false,
xContentTypeOptions: headers['x-content-type-options'] ? true : false, xContentTypeOptions: headers['x-content-type-options'] ? true : false,
xXSSProtection: headers['x-xss-protection'] ? true : false, xXSSProtection: headers['x-xss-protection'] ? true : false,
@@ -22,5 +22,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -66,6 +66,5 @@ return new Promise((resolve, reject) => {
}); });
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const axios = require('axios'); const axios = require('axios');
const cheerio = require('cheerio'); const cheerio = require('cheerio');
const urlLib = require('url'); const urlLib = require('url');
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const handler = async (url) => { const handler = async (url) => {
const response = await axios.get(url); const response = await axios.get(url);
@@ -45,5 +45,4 @@ const handler = async (url) => {
return { internal: internalLinks, external: externalLinks }; return { internal: internalLinks, external: externalLinks };
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const dns = require('dns').promises; const dns = require('dns').promises;
const URL = require('url-parse'); const URL = require('url-parse');
@@ -54,11 +54,6 @@ const handler = async (url, event, context) => {
if (yahooMx.length > 0) { if (yahooMx.length > 0) {
mailServices.push({ provider: 'Yahoo', value: yahooMx[0].exchange }); mailServices.push({ provider: 'Yahoo', value: yahooMx[0].exchange });
} }
// Check MX records for Mimecast
const mimecastMx = mxRecords.filter(record => record.exchange.includes('mimecast.com'));
if (mimecastMx.length > 0) {
mailServices.push({ provider: 'Mimecast', value: mimecastMx[0].exchange });
}
return { return {
mxRecords, mxRecords,
@@ -77,5 +72,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); module.exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -73,12 +73,17 @@ const handler = async (url, event, context) => {
return errorResponse('The function timed out before completing.'); return errorResponse('The function timed out before completing.');
} }
return { openPorts, failedPorts }; return {
statusCode: 200,
body: JSON.stringify({ openPorts, failedPorts }),
};
}; };
const errorResponse = (message, statusCode = 444) => { const errorResponse = (message, statusCode = 444) => {
return { error: message }; return {
statusCode: statusCode,
body: JSON.stringify({ error: message }),
};
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -4,19 +4,19 @@ const middleware = require('./_common/middleware');
const handler = async (url, event, context) => { const handler = async (url, event, context) => {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY; const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!apiKey) { if (!url) {
throw new Error( throw new Error('URL param is required');
'Missing Google API. You need to set the `GOOGLE_CLOUD_API_KEY` environment variable'
);
} }
const endpoint = `https://www.googleapis.com/pagespeedonline/v5/runPagespeed?` if (!apiKey) {
+ `url=${encodeURIComponent(url)}&category=PERFORMANCE&category=ACCESSIBILITY` throw new Error('API key (GOOGLE_CLOUD_API_KEY) not set');
+ `&category=BEST_PRACTICES&category=SEO&category=PWA&strategy=mobile` }
+ `&key=${apiKey}`;
return (await axios.get(endpoint)).data; const endpoint = `https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=${encodeURIComponent(url)}&category=PERFORMANCE&category=ACCESSIBILITY&category=BEST_PRACTICES&category=SEO&category=PWA&strategy=mobile&key=${apiKey}`;
const response = await axios.get(endpoint);
return response.data;
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -10,7 +10,7 @@ const handler = async (url) => {
{ auth: { username: process.env.TRANCO_USERNAME, password: process.env.TRANCO_API_KEY } } { auth: { username: process.env.TRANCO_USERNAME, password: process.env.TRANCO_API_KEY } }
: {}; : {};
const response = await axios.get( const response = await axios.get(
`https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 5000 }, auth, `https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 2000 }, auth,
); );
if (!response.data || !response.data.ranks || response.data.ranks.length === 0) { if (!response.data || !response.data.ranks || response.data.ranks.length === 0) {
return { skipped: `Skipping, as ${domain} isn't ranked in the top 100 million sites yet.`}; return { skipped: `Skipping, as ${domain} isn't ranked in the top 100 million sites yet.`};
@@ -21,6 +21,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -24,6 +24,4 @@ const handler = async (url) => {
}; };
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -67,5 +67,4 @@ const handler = async function(url) {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const puppeteer = require('puppeteer-core');
const chromium = require('chrome-aws-lambda'); const chromium = require('chrome-aws-lambda');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (targetUrl) => { const screenshotHandler = async (targetUrl) => {
if (!targetUrl) { if (!targetUrl) {
throw new Error('URL is missing from queryStringParameters'); throw new Error('URL is missing from queryStringParameters');
@@ -21,7 +21,7 @@ const handler = async (targetUrl) => {
let browser = null; let browser = null;
try { try {
browser = await puppeteer.launch({ browser = await puppeteer.launch({
args: [...chromium.args, '--no-sandbox'], // Add --no-sandbox flag args: chromium.args,
defaultViewport: { width: 800, height: 600 }, defaultViewport: { width: 800, height: 600 },
executablePath: process.env.CHROME_PATH || await chromium.executablePath, executablePath: process.env.CHROME_PATH || await chromium.executablePath,
headless: chromium.headless, headless: chromium.headless,
@@ -58,5 +58,4 @@ const handler = async (targetUrl) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(screenshotHandler);
module.exports.handler = middleware(handler);

View File

@@ -38,7 +38,7 @@ const isPgpSigned = (result) => {
return false; return false;
}; };
const handler = async (urlParam) => { const securityTxtHandler = async (urlParam) => {
let url; let url;
try { try {
@@ -69,6 +69,8 @@ const handler = async (urlParam) => {
return { isPresent: false }; return { isPresent: false };
}; };
exports.handler = middleware(securityTxtHandler);
async function fetchSecurityTxt(baseURL, path) { async function fetchSecurityTxt(baseURL, path) {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const url = new URL(path, baseURL); const url = new URL(path, baseURL);
@@ -89,6 +91,3 @@ async function fetchSecurityTxt(baseURL, path) {
}); });
}); });
} }
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const axios = require('axios'); const axios = require('axios');
const xml2js = require('xml2js'); const xml2js = require('xml2js');
@@ -6,17 +6,15 @@ const xml2js = require('xml2js');
const handler = async (url) => { const handler = async (url) => {
let sitemapUrl = `${url}/sitemap.xml`; let sitemapUrl = `${url}/sitemap.xml`;
const hardTimeOut = 5000;
try { try {
// Try to fetch sitemap directly // Try to fetch sitemap directly
let sitemapRes; let sitemapRes;
try { try {
sitemapRes = await axios.get(sitemapUrl, { timeout: hardTimeOut }); sitemapRes = await axios.get(sitemapUrl, { timeout: 5000 });
} catch (error) { } catch (error) {
if (error.response && error.response.status === 404) { if (error.response && error.response.status === 404) {
// If sitemap not found, try to fetch it from robots.txt // If sitemap not found, try to fetch it from robots.txt
const robotsRes = await axios.get(`${url}/robots.txt`, { timeout: hardTimeOut }); const robotsRes = await axios.get(`${url}/robots.txt`, { timeout: 5000 });
const robotsTxt = robotsRes.data.split('\n'); const robotsTxt = robotsRes.data.split('\n');
for (let line of robotsTxt) { for (let line of robotsTxt) {
@@ -27,10 +25,13 @@ const handler = async (url) => {
} }
if (!sitemapUrl) { if (!sitemapUrl) {
return { skipped: 'No sitemap found' }; return {
statusCode: 404,
body: JSON.stringify({ skipped: 'No sitemap found' }),
};
} }
sitemapRes = await axios.get(sitemapUrl, { timeout: hardTimeOut }); sitemapRes = await axios.get(sitemapUrl, { timeout: 5000 });
} else { } else {
throw error; // If other error, throw it throw error; // If other error, throw it
} }
@@ -39,16 +40,25 @@ const handler = async (url) => {
const parser = new xml2js.Parser(); const parser = new xml2js.Parser();
const sitemap = await parser.parseStringPromise(sitemapRes.data); const sitemap = await parser.parseStringPromise(sitemapRes.data);
return sitemap; return {
statusCode: 200,
body: JSON.stringify(sitemap),
};
} catch (error) { } catch (error) {
// If error occurs
console.log(error.message);
if (error.code === 'ECONNABORTED') { if (error.code === 'ECONNABORTED') {
return { error: `Request timed-out after ${hardTimeOut}ms` }; return {
statusCode: 500,
body: JSON.stringify({ error: 'Request timed out' }),
};
} else { } else {
return { error: error.message }; return {
statusCode: 500,
body: JSON.stringify({ error: error.message }),
};
} }
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,4 +1,4 @@
const middleware = require('./_common/middleware'); const commonMiddleware = require('./_common/middleware');
const axios = require('axios'); const axios = require('axios');
const cheerio = require('cheerio'); const cheerio = require('cheerio');
@@ -50,9 +50,16 @@ const handler = async (url) => {
}; };
if (Object.keys(metadata).length === 0) { if (Object.keys(metadata).length === 0) {
return { skipped: 'No metadata found' }; return {
statusCode: 200,
body: JSON.stringify({ skipped: 'No metadata found' }),
};
} }
return metadata;
return {
statusCode: 200,
body: JSON.stringify(metadata),
};
} catch (error) { } catch (error) {
return { return {
statusCode: 500, statusCode: 500,
@@ -61,5 +68,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = commonMiddleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const tls = require('tls'); const tls = require('tls');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (urlString) => { const fetchSiteCertificateHandler = async (urlString) => {
try { try {
const parsedUrl = new URL(urlString); const parsedUrl = new URL(urlString);
const options = { const options = {
@@ -40,5 +40,4 @@ const handler = async (urlString) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(fetchSiteCertificateHandler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const https = require('https');
const { performance, PerformanceObserver } = require('perf_hooks'); const { performance, PerformanceObserver } = require('perf_hooks');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const checkURLHandler = async (url) => {
if (!url) { if (!url) {
throw new Error('You must provide a URL query parameter!'); throw new Error('You must provide a URL query parameter!');
} }
@@ -55,5 +55,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(checkURLHandler);
module.exports.handler = middleware(handler);

View File

@@ -1,7 +1,7 @@
const Wappalyzer = require('wappalyzer'); const Wappalyzer = require('wappalyzer');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (url) => { const analyzeSiteTechnologies = async (url) => {
const options = {}; const options = {};
const wappalyzer = new Wappalyzer(options); const wappalyzer = new Wappalyzer(options);
@@ -27,5 +27,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(analyzeSiteTechnologies);
module.exports.handler = middleware(handler);

View File

@@ -5,9 +5,6 @@ const middleware = require('./_common/middleware');
const getGoogleSafeBrowsingResult = async (url) => { const getGoogleSafeBrowsingResult = async (url) => {
try { try {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY; const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!apiKey) {
return { error: 'GOOGLE_CLOUD_API_KEY is required for the Google Safe Browsing check' };
}
const apiEndpoint = `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`; const apiEndpoint = `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`;
const requestBody = { const requestBody = {
@@ -66,15 +63,11 @@ const getPhishTankResult = async (url) => {
} }
const getCloudmersiveResult = async (url) => { const getCloudmersiveResult = async (url) => {
const apiKey = process.env.CLOUDMERSIVE_API_KEY;
if (!apiKey) {
return { error: 'CLOUDMERSIVE_API_KEY is required for the Cloudmersive check' };
}
try { try {
const endpoint = 'https://api.cloudmersive.com/virus/scan/website'; const endpoint = 'https://api.cloudmersive.com/virus/scan/website';
const headers = { const headers = {
'Content-Type': 'application/x-www-form-urlencoded', 'Content-Type': 'application/x-www-form-urlencoded',
'Apikey': apiKey, 'Apikey': process.env.CLOUDMERSIVE_API_KEY,
}; };
const data = `Url=${encodeURIComponent(url)}`; const data = `Url=${encodeURIComponent(url)}`;
const response = await axios.post(endpoint, data, { headers }); const response = await axios.post(endpoint, data, { headers });
@@ -99,5 +92,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -25,5 +25,4 @@ const handler = async (url) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -2,7 +2,7 @@ const traceroute = require('traceroute');
const url = require('url'); const url = require('url');
const middleware = require('./_common/middleware'); const middleware = require('./_common/middleware');
const handler = async (urlString, context) => { const executeTraceroute = async (urlString, context) => {
// Parse the URL and get the hostname // Parse the URL and get the hostname
const urlObject = url.parse(urlString); const urlObject = url.parse(urlString);
const host = urlObject.hostname; const host = urlObject.hostname;
@@ -28,5 +28,4 @@ const handler = async (urlString, context) => {
}; };
}; };
module.exports = middleware(handler); exports.handler = middleware(executeTraceroute);
module.exports.handler = middleware(handler);

View File

@@ -29,5 +29,4 @@ const handler = async (url, event, context) => {
} }
}; };
module.exports = middleware(handler); exports.handler = middleware(handler);
module.exports.handler = middleware(handler);

View File

@@ -83,7 +83,7 @@ const fetchFromMyAPI = async (hostname) => {
} }
}; };
const handler = async (url) => { const fetchWhoisData = async (url) => {
if (!url.startsWith('http://') && !url.startsWith('https://')) { if (!url.startsWith('http://') && !url.startsWith('https://')) {
url = 'http://' + url; url = 'http://' + url;
} }
@@ -106,6 +106,4 @@ const handler = async (url) => {
}; };
}; };
module.exports = middleware(handler); exports.handler = middleware(fetchWhoisData);
module.exports.handler = middleware(handler);

View File

@@ -5,7 +5,7 @@
publish = "build" publish = "build"
functions = "api" functions = "api"
# Environmental variables and optional secrets # Environmental variables and optioanl secrets
[build.environment] [build.environment]
# Build configuration env vars (uncomment if you want to conigure these) # Build configuration env vars (uncomment if you want to conigure these)
# CI="false" # Set CI to false, to prevent warnings from exiting the build # CI="false" # Set CI to false, to prevent warnings from exiting the build

View File

@@ -1,10 +1,10 @@
{ {
"name": "web-check", "name": "web-check",
"version": "1.1.2", "version": "1.0.0",
"private": false, "private": false,
"description": "All-in-one OSINT tool for analyzing any website", "description": "All-in-one OSINT tool for analyzing any website",
"repository": "github:lissy93/web-check", "repository": "github:lissy93/web-check",
"homepage": "https://web-check.xyz", "homepage": "https://web-check.as93.net",
"license": "MIT", "license": "MIT",
"author": { "author": {
"name": "Alicia Sykes", "name": "Alicia Sykes",
@@ -20,9 +20,7 @@
"start": "react-scripts start", "start": "react-scripts start",
"build": "react-scripts build", "build": "react-scripts build",
"test": "react-scripts test", "test": "react-scripts test",
"dev:vercel": "PLATFORM='vercel' npx vercel dev", "eject": "react-scripts eject"
"dev:netlify": "PLATFORM='netlify' npx netlify dev",
"dev:node": "npx concurrently --names \"frontend,backend\" \"REACT_APP_API_ENDPOINT='http://localhost:3001/api' npx react-scripts start\" \"PLATFORM='node' DISABLE_GUI='true' PORT='3001' API_CORS_ORIGIN='*' npx nodemon server\""
}, },
"dependencies": { "dependencies": {
"@netlify/functions": "^1.6.0", "@netlify/functions": "^1.6.0",
@@ -37,15 +35,14 @@
"@types/react-router-dom": "^5.3.3", "@types/react-router-dom": "^5.3.3",
"@types/react-simple-maps": "^3.0.0", "@types/react-simple-maps": "^3.0.0",
"@types/styled-components": "^5.1.26", "@types/styled-components": "^5.1.26",
"aws-serverless-express": "^3.4.0",
"axios": "^1.4.0", "axios": "^1.4.0",
"cheerio": "^1.0.0-rc.12", "cheerio": "^1.0.0-rc.12",
"chrome-aws-lambda": "^10.1.0", "chrome-aws-lambda": "^10.1.0",
"chromium": "^3.0.3", "chromium": "^3.0.3",
"connect-history-api-fallback": "^2.0.0", "connect-history-api-fallback": "^2.0.0",
"cors": "^2.8.5",
"csv-parser": "^3.0.0", "csv-parser": "^3.0.0",
"dotenv": "^16.3.1", "dotenv": "^16.3.1",
"express-rate-limit": "^7.2.0",
"flatted": "^3.2.7", "flatted": "^3.2.7",
"follow-redirects": "^1.15.2", "follow-redirects": "^1.15.2",
"got": "^13.0.0", "got": "^13.0.0",

View File

@@ -1,39 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p class="moji">😪</p>
<p>There was an error finding this route.</p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
.moji { font-size: 8rem; margin: 0; }
</style>
</body>
</html>

View File

@@ -18,11 +18,8 @@
<meta property="og:title" content="Web Check"> <meta property="og:title" content="Web Check">
<meta property="og:description" content="All-in-one Website OSINT Scanner"> <meta property="og:description" content="All-in-one Website OSINT Scanner">
<meta property="og:image" content="/banner.png"> <meta property="og:image" content="/banner.png">
<meta property="og:url" content="https://web-check.xyz"> <meta property="og:url" content="https://web-check.as93.net">
<meta name="twitter:card" content="summary_large_image"> <meta name="twitter:card" content="summary_large_image">
<!-- DarkReader -->
<meta name="darkreader-lock">
</head> </head>
<body> <body>
<noscript> <noscript>

View File

@@ -1,37 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p><!-- CONTENT --></p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
</style>
</body>
</html>

152
server.js
View File

@@ -1,111 +1,75 @@
const express = require('express'); const express = require('express');
const awsServerlessExpress = require('aws-serverless-express');
const fs = require('fs'); const fs = require('fs');
const path = require('path'); const path = require('path');
const cors = require('cors');
const rateLimit = require('express-rate-limit');
const historyApiFallback = require('connect-history-api-fallback'); const historyApiFallback = require('connect-history-api-fallback');
require('dotenv').config(); require('dotenv').config();
const app = express(); const app = express();
const port = process.env.PORT || 3000; // The port to run the server on
const API_DIR = '/api'; // Name of the dir containing the lambda functions const API_DIR = '/api'; // Name of the dir containing the lambda functions
const dirPath = path.join(__dirname, API_DIR); // Path to the lambda functions dir const dirPath = path.join(__dirname, API_DIR); // Path to the lambda functions dir
const guiPath = path.join(__dirname, 'build'); const guiPath = path.join(__dirname, 'build');
const placeholderFilePath = path.join(__dirname, 'public', 'placeholder.html');
const handlers = {}; // Will store list of API endpoints
process.env.WC_SERVER = 'true'; // Tells middleware to return in non-lambda mode
// Enable CORS // Execute the lambda function
app.use(cors({ const executeHandler = async (handler, req) => {
origin: process.env.API_CORS_ORIGIN || '*', return new Promise((resolve, reject) => {
})); const callback = (err, response) => err ? reject(err) : resolve(response);
const promise = handler(req, {}, callback);
// Define max requests within each time frame if (promise && typeof promise.then === 'function') {
const limits = [ promise.then(resolve).catch(reject);
{ timeFrame: 10 * 60, max: 100, messageTime: '10 minutes' }, }
{ timeFrame: 60 * 60, max: 250, messageTime: '1 hour' }, });
{ timeFrame: 12 * 60 * 60, max: 500, messageTime: '12 hours' },
];
// Construct a message to be returned if the user has been rate-limited
const makeLimiterResponseMsg = (retryAfter) => {
const why = 'This keeps the service running smoothly for everyone. '
+ 'You can get around these limits by running your own instance of Web Check.';
return `You've been rate-limited, please try again in ${retryAfter} seconds.\n${why}`;
}; };
// Create rate limiters for each time frame // Array of all the lambda function file names
const limiters = limits.map(limit => rateLimit({ const fileNames = fs.readdirSync(dirPath, { withFileTypes: true })
windowMs: limit.timeFrame * 1000,
max: limit.max,
standardHeaders: true,
legacyHeaders: false,
message: { error: makeLimiterResponseMsg(limit.messageTime) }
}));
// If rate-limiting enabled, then apply the limiters to the /api endpoint
if (process.env.API_ENABLE_RATE_LIMIT === 'true') {
app.use('/api', limiters);
}
// Read and register each API function as an Express routes
fs.readdirSync(dirPath, { withFileTypes: true })
.filter(dirent => dirent.isFile() && dirent.name.endsWith('.js')) .filter(dirent => dirent.isFile() && dirent.name.endsWith('.js'))
.forEach(dirent => { .map(dirent => dirent.name);
const routeName = dirent.name.split('.')[0];
const handlers = {};
fileNames.forEach(file => {
const routeName = file.split('.')[0];
const route = `${API_DIR}/${routeName}`; const route = `${API_DIR}/${routeName}`;
const handler = require(path.join(dirPath, dirent.name)); const handler = require(path.join(dirPath, file)).handler;
handlers[route] = handler; handlers[route] = handler;
app.get(route, async (req, res) => { app.get(route, async (req, res) => {
try { try {
await handler(req, res); const { statusCode = 200, body = '' } = await executeHandler(handler, req);
res.status(statusCode).json(JSON.parse(body));
} catch (err) { } catch (err) {
res.status(500).json({ error: err.message }); res.status(500).json({ error: err.message });
} }
}); });
}); });
// Create a single API endpoint to execute all lambda functions
app.get('/api', async (req, res) => {
const results = {};
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 20000;
const executeHandler = async (handler, req, res) => {
return new Promise(async (resolve, reject) => {
try {
const mockRes = {
status: (statusCode) => mockRes,
json: (body) => resolve({ body }),
};
await handler({ ...req, query: { url } }, mockRes);
} catch (err) {
reject(err);
}
});
};
const timeout = (ms, jobName = null) => { const timeout = (ms, jobName = null) => {
return new Promise((_, reject) => { return new Promise((_, reject) => {
setTimeout(() => { setTimeout(() => {
reject(new Error( reject(new Error(`Timed out after the ${ms/1000} second limit${jobName ? `, when executing the ${jobName} task` : ''}`));
`Timed out after ${ms/1000} seconds${jobName ? `, when executing ${jobName}` : ''}`
));
}, ms); }, ms);
}); });
}; }
app.get('/api', async (req, res) => {
const results = {};
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 15000;
const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => { const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => {
const routeName = route.replace(`${API_DIR}/`, ''); const routeName = route.replace(`${API_DIR}/`, '');
try { try {
const result = await Promise.race([ const result = await Promise.race([
executeHandler(handler, req, res), executeHandler(handler, { query: { url } }),
timeout(maxExecutionTime, routeName) timeout(maxExecutionTime, routeName)
]); ]);
results[routeName] = result.body; results[routeName] = JSON.parse((result || {}).body);
} catch (err) { } catch (err) {
results[routeName] = { error: err.message }; results[routeName] = { error: err.message };
} }
@@ -118,59 +82,35 @@ fs.readdirSync(dirPath, { withFileTypes: true })
// Handle SPA routing // Handle SPA routing
app.use(historyApiFallback({ app.use(historyApiFallback({
rewrites: [ rewrites: [
{ from: /^\/api\/.*$/, to: (context) => context.parsedUrl.path }, { from: /^\/api\/.*$/, to: function(context) { return context.parsedUrl.path; } },
] ]
})); }));
// Serve up the GUI - if build dir exists, and GUI feature enabled // Serve up the GUI - if build dir exists, and GUI feature enabled
if (process.env.DISABLE_GUI && process.env.DISABLE_GUI !== 'false') { if (process.env.DISABLE_GUI && process.env.DISABLE_GUI !== 'false') {
app.get('*', async (req, res) => { app.get('*', (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8'); res.status(500).send(
const htmlContent = placeholderContent.replace( 'Welcome to Web-Check!<br />Access the API endpoints at '
'<!-- CONTENT -->',
'Web-Check API is up and running!<br />Access the endpoints at '
+'<a href="/api"><code>/api</code></a>' +'<a href="/api"><code>/api</code></a>'
); );
res.status(500).send(htmlContent);
}); });
} else if (!fs.existsSync(guiPath)) { } else if (!fs.existsSync(guiPath)) {
app.get('*', async (req, res) => { app.get('*', (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8'); res.status(500).send(
const htmlContent = placeholderContent.replace( 'Welcome to Web-Check!<br />Looks like the GUI app has not yet been compiled, '
'<!-- CONTENT -->', +'run <code>yarn build</code> to continue, then restart the server.'
'Looks like the GUI app has not yet been compiled.<br /> ' +
'Run <code>yarn build</code> to continue, then restart the server.'
); );
res.status(500).send(htmlContent);
}); });
} else { // GUI enabled, and build files present, let's go!! } else { // GUI enabled, and build files present, let's go!!
app.use(express.static(guiPath)); app.use(express.static(guiPath));
} }
app.use((req, res, next) => { // Create serverless express server
res.status(404).sendFile(path.join(__dirname, 'public', 'error.html')); const port = process.env.PORT || 3000;
const server = awsServerlessExpress.createServer(app).listen(port, () => {
console.log(`Server is running on port ${port}`);
}); });
// Print nice welcome message to user exports.handler = (event, context) => {
const printMessage = () => { awsServerlessExpress.proxy(server, event, context);
console.log(
`\x1b[36m\n` +
' __ __ _ ___ _ _ \n' +
' \\ \\ / /__| |__ ___ / __| |_ ___ __| |__\n' +
' \\ \\/\\/ / -_) \'_ \\___| (__| \' \\/ -_) _| / /\n' +
' \\_/\\_/\\___|_.__/ \\___|_||_\\___\\__|_\\_\\\n' +
`\x1b[0m\n`,
`\x1b[1m\x1b[32m🚀 Web-Check is up and running at http://localhost:${port} \x1b[0m\n\n`,
`\x1b[2m\x1b[36m🛟 For documentation and support, visit the GitHub repo: ` +
`https://github.com/lissy93/web-check \n`,
`💖 Found Web-Check useful? Consider sponsoring us on GitHub ` +
`to help fund maintenance & development.\x1b[0m`
);
}; };
// Create server
app.listen(port, () => {
printMessage();
});

View File

@@ -4,214 +4,121 @@ provider:
name: aws name: aws
runtime: nodejs14.x runtime: nodejs14.x
region: us-east-1 region: us-east-1
# environment:
# GOOGLE_CLOUD_API_KEY: ${ssm:GOOGLE_CLOUD_API_KEY~true, ''}
# TORRENT_IP_API_KEY: ${ssm:TORRENT_IP_API_KEY~true, ''}
# SECURITY_TRAILS_API_KEY: ${ssm:SECURITY_TRAILS_API_KEY~true, ''}
# BUILT_WITH_API_KEY: ${ssm:BUILT_WITH_API_KEY~true, ''}
# URL_SCAN_API_KEY: ${ssm:URL_SCAN_API_KEY~true, ''}
# TRANCO_USERNAME: ${ssm:TRANCO_USERNAME~true, ''}
# TRANCO_API_KEY: ${ssm:TRANCO_API_KEY~true, ''}
# CLOUDMERSIVE_API_KEY: ${ssm:CLOUDMERSIVE_API_KEY~true, ''}
# CHROME_PATH: ${ssm:CHROME_PATH~true, ''}
# API_TIMEOUT_LIMIT: ${ssm:API_TIMEOUT_LIMIT~true, ''}
# API_CORS_ORIGIN: ${ssm:API_CORS_ORIGIN~true, ''}
iamRoleStatements:
- Effect: Allow
Action:
- ssm:GetParameter
Resource:
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/GOOGLE_CLOUD_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TORRENT_IP_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/SECURITY_TRAILS_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/BUILT_WITH_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/URL_SCAN_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TRANCO_USERNAME
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/TRANCO_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/CLOUDMERSIVE_API_KEY
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/CHROME_PATH
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/API_TIMEOUT_LIMIT
- arn:aws:ssm:us-east-1:${env:AWS_ACCOUNT_ID}:parameter/API_CORS_ORIGIN
functions: functions:
archives:
handler: api/archives.handler
events:
- http:
path: api/archives
method: get
blockLists:
handler: api/block-lists.handler
events:
- http:
path: api/block-lists
method: get
carbon:
handler: api/carbon.handler
events:
- http:
path: api/carbon
method: get
cookies:
handler: api/cookies.handler
events:
- http:
path: api/cookies
method: get
dnsServer:
handler: api/dns-server.handler
events:
- http:
path: api/dns-server
method: get
dns:
handler: api/dns.handler
events:
- http:
path: api/dns
method: get
dnssec: dnssec:
handler: api/dnssec.handler handler: api/dnssec.handler
events: events:
- http: - http:
path: api/dnssec path: api/dnssec
method: get method: get
features:
handler: api/features.handler
events:
- http:
path: api/features
method: get
firewall:
handler: api/firewall.handler
events:
- http:
path: api/firewall
method: get
getIp:
handler: api/get-ip.handler
events:
- http:
path: api/get-ip
method: get
headers:
handler: api/headers.handler
events:
- http:
path: api/headers
method: get
hsts:
handler: api/hsts.handler
events:
- http:
path: api/hsts
method: get
httpSecurity:
handler: api/http-security.handler
events:
- http:
path: api/http-security
method: get
legacyRank:
handler: api/legacy-rank.handler
events:
- http:
path: api/legacy-rank
method: get
linkedPages: linkedPages:
handler: api/linked-pages.handler handler: api/linked-pages.handler
events: events:
- http: - http:
path: api/linked-pages path: api/linked-pages
method: get method: get
mailConfig:
handler: api/mail-config.handler
events:
- http:
path: api/mail-config
method: get
ports:
handler: api/ports.handler
events:
- http:
path: api/ports
method: get
quality:
handler: api/quality.handler
events:
- http:
path: api/quality
method: get
rank:
handler: api/rank.handler
events:
- http:
path: api/rank
method: get
redirects:
handler: api/redirects.handler
events:
- http:
path: api/redirects
method: get
robotsTxt: robotsTxt:
handler: api/robots-txt.handler handler: api/robots-txt.handler
events: events:
- http: - http:
path: api/robots-txt path: api/robots-txt
method: get method: get
screenshot:
handler: api/screenshot.handler
events:
- http:
path: api/screenshot
method: get
securityTxt:
handler: api/security-txt.handler
events:
- http:
path: api/security-txt
method: get
sitemap:
handler: api/sitemap.handler
events:
- http:
path: api/sitemap
method: get
socialTags:
handler: api/social-tags.handler
events:
- http:
path: api/social-tags
method: get
ssl: ssl:
handler: api/ssl.handler handler: api/ssl.handler
events: events:
- http: - http:
path: api/ssl path: api/ssl
method: get method: get
whois:
handler: api/whois.handler
events:
- http:
path: api/whois
method: get
carbon:
handler: api/carbon.handler
events:
- http:
path: api/carbon
method: get
features:
handler: api/features.handler
events:
- http:
path: api/features
method: get
mailConfig:
handler: api/mail-config.handler
events:
- http:
path: api/mail-config
method: get
screenshot:
handler: api/screenshot.handler
events:
- http:
path: api/screenshot
method: get
status: status:
handler: api/status.handler handler: api/status.handler
events: events:
- http: - http:
path: api/status path: api/status
method: get method: get
cookies:
handler: api/cookies.handler
events:
- http:
path: api/cookies
method: get
getIp:
handler: api/get-ip.handler
events:
- http:
path: api/get-ip
method: get
ports:
handler: api/ports.handler
events:
- http:
path: api/ports
method: get
securityTxt:
handler: api/security-txt.handler
events:
- http:
path: api/security-txt
method: get
techStack: techStack:
handler: api/tech-stack.handler handler: api/tech-stack.handler
events: events:
- http: - http:
path: api/tech-stack path: api/tech-stack
method: get method: get
threats: dnsServer:
handler: api/threats.handler handler: api/dns-server.handler
events: events:
- http: - http:
path: api/threats path: api/dns-server
method: get method: get
tls: headers:
handler: api/tls.handler handler: api/headers.handler
events: events:
- http: - http:
path: api/tls path: api/headers
method: get
quality:
handler: api/quality.handler
events:
- http:
path: api/quality
method: get
sitemap:
handler: api/sitemap.handler
events:
- http:
path: api/sitemap
method: get method: get
traceRoute: traceRoute:
handler: api/trace-route.handler handler: api/trace-route.handler
@@ -219,38 +126,54 @@ functions:
- http: - http:
path: api/trace-route path: api/trace-route
method: get method: get
dns:
handler: api/dns.handler
events:
- http:
path: api/dns
method: get
hsts:
handler: api/hsts.handler
events:
- http:
path: api/hsts
method: get
redirects:
handler: api/redirects.handler
events:
- http:
path: api/redirects
method: get
socialTags:
handler: api/social-tags.handler
events:
- http:
path: api/social-tags
method: get
txtRecords: txtRecords:
handler: api/txt-records.handler handler: api/txt-records.handler
events: events:
- http: - http:
path: api/txt-records path: api/txt-records
method: get method: get
whois:
handler: api/whois.handler
events:
- http:
path: api/whois
method: get
plugins: plugins:
- serverless-webpack # - serverless-webpack
# - serverless-domain-manager - serverless-domain-manager
# - serverless-offline - serverless-offline
custom: custom:
webpack: webpack:
webpackConfig: 'api/_common/aws-webpack.config.js' webpackConfig: 'api/_common/aws-webpack.config.js'
includeModules: false includeModules: true
packagerOptions:
noInstall: true
# customDomain: customDomain:
# domainName: example.com domainName: example.com
# basePath: 'api' basePath: 'api'
# stage: ${self:provider.stage} stage: ${self:provider.stage}
# createRoute53Record: true createRoute53Record: true
# serverless-offline: serverless-offline:
# prefix: '' prefix: ''
# httpPort: 3000 httpPort: 3000

View File

@@ -12,6 +12,7 @@ const Container = Styled.main`
background: ${colors.background}; background: ${colors.background};
color: ${colors.textColor}; color: ${colors.textColor};
width: 100vw; width: 100vw;
height: 100vh;
margin: 0; margin: 0;
`; `;

View File

@@ -159,7 +159,7 @@ export const ExpandableRow = (props: RowProps) => {
{ rowList?.map((row: RowProps, index: number) => { { rowList?.map((row: RowProps, index: number) => {
return ( return (
<SubRow key={`${row.lbl}-${index}`}> <SubRow key={`${row.lbl}-${index}`}>
<span className="lbl" title={row.title?.toString()}>{row.lbl}</span> <span className="lbl" title={row.title}>{row.lbl}</span>
<span className="val" title={row.val} onClick={() => copyToClipboard(row.val)}> <span className="val" title={row.val} onClick={() => copyToClipboard(row.val)}>
{formatValue(row.val)} {formatValue(row.val)}
</span> </span>
@@ -199,7 +199,7 @@ const Row = (props: RowProps) => {
if (children) return <StyledRow key={`${lbl}-${val}`}>{children}</StyledRow>; if (children) return <StyledRow key={`${lbl}-${val}`}>{children}</StyledRow>;
return ( return (
<StyledRow key={`${lbl}-${val}`}> <StyledRow key={`${lbl}-${val}`}>
{ lbl && <span className="lbl" title={title?.toString()}>{lbl}</span> } { lbl && <span className="lbl" title={title}>{lbl}</span> }
<span className="val" title={val} onClick={() => copyToClipboard(val)}> <span className="val" title={val} onClick={() => copyToClipboard(val)}>
{formatValue(val)} {formatValue(val)}
</span> </span>

View File

@@ -6,13 +6,11 @@ const BlockListsCard = (props: {data: any, title: string, actionButtons: any }):
const blockLists = props.data.blocklists; const blockLists = props.data.blocklists;
return ( return (
<Card heading={props.title} actionButtons={props.actionButtons}> <Card heading={props.title} actionButtons={props.actionButtons}>
{ blockLists.map((blocklist: any, blockIndex: number) => ( { blockLists.map((blocklist: any) => (
<Row <Row
title={blocklist.serverIp} title={blocklist.serverIp}
lbl={blocklist.server} lbl={blocklist.server}
val={blocklist.isBlocked ? '❌ Blocked' : '✅ Not Blocked'} val={blocklist.isBlocked ? '❌ Blocked' : '✅ Not Blocked'} />
key={`blocklist-${blockIndex}-${blocklist.serverIp}`}
/>
))} ))}
</Card> </Card>
); );

View File

@@ -18,12 +18,12 @@ const DnsServerCard = (props: {data: any, title: string, actionButtons: any }):
return ( return (
<Card heading={props.title} actionButtons={props.actionButtons} styles={cardStyles}> <Card heading={props.title} actionButtons={props.actionButtons} styles={cardStyles}>
{dnsSecurity.dns.map((dns: any, index: number) => { {dnsSecurity.dns.map((dns: any, index: number) => {
return (<div key={`dns-${index}`}> return (<>
{ dnsSecurity.dns.length > 1 && <Heading as="h4" size="small" color={colors.primary}>DNS Server #{index+1}</Heading> } { dnsSecurity.dns.length > 1 && <Heading as="h4" size="small" color={colors.primary}>DNS Server #{index+1}</Heading> }
<Row lbl="IP Address" val={dns.address} key={`ip-${index}`} /> <Row lbl="IP Address" val={dns.address} key={`ip-${index}`} />
{ dns.hostname && <Row lbl="Hostname" val={dns.hostname} key={`host-${index}`} /> } { dns.hostname && <Row lbl="Hostname" val={dns.hostname} key={`host-${index}`} /> }
<Row lbl="DoH Support" val={dns.dohDirectSupports ? '✅ Yes*' : '❌ No*'} key={`doh-${index}`} /> <Row lbl="DoH Support" val={dns.dohDirectSupports ? '✅ Yes*' : '❌ No*'} key={`doh-${index}`} />
</div>); </>);
})} })}
{dnsSecurity.dns.length > 0 && (<small> {dnsSecurity.dns.length > 0 && (<small>
* DoH Support is determined by the DNS server's response to a DoH query. * DoH Support is determined by the DNS server's response to a DoH query.

View File

@@ -13,7 +13,7 @@ span.val {
const ServerStatusCard = (props: { data: any, title: string, actionButtons: any }): JSX.Element => { const ServerStatusCard = (props: { data: any, title: string, actionButtons: any }): JSX.Element => {
const serverStatus = props.data; const serverStatus = props.data;
return ( return (
<Card heading={props.title.toString()} actionButtons={props.actionButtons} styles={cardStyles}> <Card heading={props.title} actionButtons={props.actionButtons} styles={cardStyles}>
<Row lbl="" val=""> <Row lbl="" val="">
<span className="lbl">Is Up?</span> <span className="lbl">Is Up?</span>
{ serverStatus.isUp ? <span className="val up"> Online</span> : <span className="val down"> Offline</span>} { serverStatus.isUp ? <span className="val up"> Online</span> : <span className="val down"> Offline</span>}

View File

@@ -90,6 +90,7 @@ const SslCertCard = (props: { data: any, title: string, actionButtons: any }): J
{ valid_from && <DataRow lbl="Renewed" val={formatDate(valid_from)} /> } { valid_from && <DataRow lbl="Renewed" val={formatDate(valid_from)} /> }
{ serialNumber && <DataRow lbl="Serial Num" val={serialNumber} /> } { serialNumber && <DataRow lbl="Serial Num" val={serialNumber} /> }
{ fingerprint && <DataRow lbl="Fingerprint" val={fingerprint} /> } { fingerprint && <DataRow lbl="Fingerprint" val={fingerprint} /> }
{ fingerprint && <DataRow lbl="Fingerprint" val={fingerprint} /> }
{ ext_key_usage && <ListRow title="Extended Key Usage" list={getExtendedKeyUsage(ext_key_usage)} /> } { ext_key_usage && <ListRow title="Extended Key Usage" list={getExtendedKeyUsage(ext_key_usage)} /> }
</Card> </Card>

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
import Button from 'components/Form/Button'; import Button from 'components/Form/Button';
import { ExpandableRow } from 'components/Form/Row'; import { ExpandableRow } from 'components/Form/Row';
@@ -29,10 +29,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [cipherSuites, setCipherSuites] = useState(makeCipherSuites(props.data)); const [cipherSuites, setCipherSuites] = useState(makeCipherSuites(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => { // Update cipher suites when data changes
setCipherSuites(makeCipherSuites(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setCipherSuites([]); setCipherSuites([]);
setLoadState('loading'); setLoadState('loading');
@@ -52,7 +48,7 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
<Card heading={props.title} actionButtons={props.actionButtons}> <Card heading={props.title} actionButtons={props.actionButtons}>
{ cipherSuites.length && cipherSuites.map((cipherSuite: any, index: number) => { { cipherSuites.length && cipherSuites.map((cipherSuite: any, index: number) => {
return ( return (
<ExpandableRow key={`tls-${index}`} lbl={cipherSuite.title} val="" rowList={cipherSuite.fields} /> <ExpandableRow lbl={cipherSuite.title} val="" rowList={cipherSuite.fields} />
); );
})} })}
{ !cipherSuites.length && ( { !cipherSuites.length && (

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
import Button from 'components/Form/Button'; import Button from 'components/Form/Button';
import { ExpandableRow } from 'components/Form/Row'; import { ExpandableRow } from 'components/Form/Row';
@@ -38,10 +38,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [clientSupport, setClientSupport] = useState(makeClientSupport(props.data)); const [clientSupport, setClientSupport] = useState(makeClientSupport(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => {
setClientSupport(makeClientSupport(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setClientSupport([]); setClientSupport([]);
setLoadState('loading'); setLoadState('loading');
@@ -59,15 +55,8 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const scanId = props.data?.id; const scanId = props.data?.id;
return ( return (
<Card heading={props.title} actionButtons={props.actionButtons}> <Card heading={props.title} actionButtons={props.actionButtons}>
{clientSupport.map((support: any, index: number) => { {clientSupport.map((support: any) => {
return ( return (<ExpandableRow lbl={support.title} val={support.value || '?'} rowList={support.fields} />)
<ExpandableRow
key={`tls-client-${index}`}
lbl={support.title}
val={support.value || '?'}
rowList={support.fields}
/>
)
})} })}
{ !clientSupport.length && ( { !clientSupport.length && (
<div> <div>

View File

@@ -1,5 +1,5 @@
import { useState, useEffect } from 'react'; import { useState } from 'react';
import styled from 'styled-components'; import styled from 'styled-components';
import colors from 'styles/colors'; import colors from 'styles/colors';
import { Card } from 'components/Form/Card'; import { Card } from 'components/Form/Card';
@@ -74,11 +74,6 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
const [tlsResults, setTlsResults] = useState(makeResults(props.data)); const [tlsResults, setTlsResults] = useState(makeResults(props.data));
const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined); const [loadState, setLoadState] = useState<undefined | 'loading' | 'success' | 'error'>(undefined);
useEffect(() => {
setTlsRowWata(makeExpandableData(props.data));
setTlsResults(makeResults(props.data));
}, [props.data]);
const updateData = (id: number) => { const updateData = (id: number) => {
setTlsRowWata([]); setTlsRowWata([]);
setLoadState('loading'); setLoadState('loading');
@@ -99,13 +94,7 @@ const TlsCard = (props: {data: any, title: string, actionButtons: any }): JSX.El
<Card heading={props.title} actionButtons={props.actionButtons}> <Card heading={props.title} actionButtons={props.actionButtons}>
{ tlsResults.length > 0 && tlsResults.map((row: any, index: number) => { { tlsResults.length > 0 && tlsResults.map((row: any, index: number) => {
return ( return (
<Row <Row lbl={row.lbl} val={row.val} plaintext={row.plaintext} listResults={row.list} />
lbl={row.lbl}
val={row.val}
plaintext={row.plaintext}
listResults={row.list}
key={`tls-issues-${index}`}
/>
); );
})} })}
<Expandable> <Expandable>

View File

@@ -92,75 +92,75 @@ const resources = [
title: 'SSL Labs Test', title: 'SSL Labs Test',
link: 'https://ssllabs.com/ssltest/analyze.html', link: 'https://ssllabs.com/ssltest/analyze.html',
icon: 'https://i.ibb.co/6bVL8JK/Qualys-ssl-labs.png', icon: 'https://i.ibb.co/6bVL8JK/Qualys-ssl-labs.png',
description: 'Analyzes the SSL configuration of a server and grades it', description: 'Analyzes the SSL configuration of a server and grades it.',
}, },
{ {
title: 'Virus Total', title: 'Virus Total',
link: 'https://virustotal.com', link: 'https://virustotal.com',
icon: 'https://i.ibb.co/dWFz0RC/Virustotal.png', icon: 'https://i.ibb.co/dWFz0RC/Virustotal.png',
description: 'Checks a URL against multiple antivirus engines', description: 'Checks a URL against multiple antivirus engines.',
searchLink: 'https://www.virustotal.com/gui/domain/{URL}', searchLink: 'https://www.virustotal.com/gui/domain/{URL}',
}, },
{ {
title: 'Shodan', title: 'Shodan',
link: 'https://shodan.io/', link: 'https://shodan.io/',
icon: 'https://i.ibb.co/SBZ8WG4/shodan.png', icon: 'https://i.ibb.co/SBZ8WG4/shodan.png',
description: 'Search engine for Internet-connected devices', description: 'Search engine for Internet-connected devices.',
searchLink: 'https://www.shodan.io/search/report?query={URL}', searchLink: 'https://www.shodan.io/search/report?query={URL}',
}, },
{ {
title: 'Archive', title: 'Archive',
link: 'https://archive.org/', link: 'https://archive.org/',
icon: 'https://i.ibb.co/nfKMvCm/Archive-org.png', icon: 'https://i.ibb.co/nfKMvCm/Archive-org.png',
description: 'View previous versions of a site via the Internet Archive', description: 'View previous versions of a site via the Internet Archive.',
searchLink: 'https://web.archive.org/web/*/{URL}', searchLink: 'https://web.archive.org/web/*/{URL}',
}, },
{ {
title: 'URLScan', title: 'URLScan',
link: 'https://urlscan.io/', link: 'https://urlscan.io/',
icon: 'https://i.ibb.co/cYXt8SH/Url-scan.png', icon: 'https://i.ibb.co/cYXt8SH/Url-scan.png',
description: 'Scans a URL and provides information about the page', description: 'Scans a URL and provides information about the page.',
}, },
{ {
title: 'Sucuri SiteCheck', title: 'Sucuri SiteCheck',
link: 'https://sitecheck.sucuri.net/', link: 'https://sitecheck.sucuri.net/',
icon: 'https://i.ibb.co/K5pTP1K/Sucuri-site-check.png', icon: 'https://i.ibb.co/K5pTP1K/Sucuri-site-check.png',
description: 'Checks a URL against blacklists and known threats', description: 'Checks a URL against blacklists and known threats.',
searchLink: 'https://www.ssllabs.com/ssltest/analyze.html?d={URL}', searchLink: 'https://www.ssllabs.com/ssltest/analyze.html?d={URL}',
}, },
{ {
title: 'Domain Tools', title: 'Domain Tools',
link: 'https://whois.domaintools.com/', link: 'https://whois.domaintools.com/',
icon: 'https://i.ibb.co/zJfCKjM/Domain-tools.png', icon: 'https://i.ibb.co/zJfCKjM/Domain-tools.png',
description: 'Run a WhoIs lookup on a domain', description: 'Run a WhoIs lookup on a domain.',
searchLink: 'https://whois.domaintools.com/{URL}', searchLink: 'https://whois.domaintools.com/{URL}',
}, },
{ {
title: 'NS Lookup', title: 'NS Lookup',
link: 'https://nslookup.io/', link: 'https://nslookup.io/',
icon: 'https://i.ibb.co/BLSWvBv/Ns-lookup.png', icon: 'https://i.ibb.co/BLSWvBv/Ns-lookup.png',
description: 'View DNS records for a domain', description: 'View DNS records for a domain.',
searchLink: 'https://www.nslookup.io/domains/{URL}/dns-records/', searchLink: 'https://www.nslookup.io/domains/{URL}/dns-records/',
}, },
{ {
title: 'DNS Checker', title: 'DNS Checker',
link: 'https://dnschecker.org/', link: 'https://dnschecker.org/',
icon: 'https://i.ibb.co/gyKtgZ1/Dns-checker.webp', icon: 'https://i.ibb.co/gyKtgZ1/Dns-checker.webp',
description: 'Check global DNS propagation across multiple servers', description: 'Check global DNS propagation across multiple servers.',
searchLink: 'https://dnschecker.org/#A/{URL}', searchLink: 'https://dnschecker.org/#A/{URL}',
}, },
{ {
title: 'Censys', title: 'Censys',
link: 'https://search.censys.io/', link: 'https://search.censys.io/',
icon: 'https://i.ibb.co/j3ZtXzM/censys.png', icon: 'https://i.ibb.co/j3ZtXzM/censys.png',
description: 'Lookup hosts associated with a domain', description: 'Lookup hosts associated with a domain.',
searchLink: 'https://search.censys.io/search?resource=hosts&q={URL}', searchLink: 'https://search.censys.io/search?resource=hosts&q={URL}',
}, },
{ {
title: 'Page Speed Insights', title: 'Page Speed Insights',
link: 'https://developers.google.com/speed/pagespeed/insights/', link: 'https://developers.google.com/speed/pagespeed/insights/',
icon: 'https://i.ibb.co/k68t9bb/Page-speed-insights.png', icon: 'https://i.ibb.co/k68t9bb/Page-speed-insights.png',
description: 'Checks the performance, accessibility and SEO of a page on mobile + desktop', description: 'Checks the performance, accessibility and SEO of a page on mobile + desktop.',
searchLink: 'https://developers.google.com/speed/pagespeed/insights/?url={URL}', searchLink: 'https://developers.google.com/speed/pagespeed/insights/?url={URL}',
}, },
{ {
@@ -204,13 +204,6 @@ const resources = [
description: 'View traffic source locations for a domain through Cloudflare', description: 'View traffic source locations for a domain through Cloudflare',
searchLink: 'https://radar.cloudflare.com/domains/domain/{URL}', searchLink: 'https://radar.cloudflare.com/domains/domain/{URL}',
}, },
{
title: 'Mozilla Observatory',
link: 'https://observatory.mozilla.org/',
icon: 'https://i.ibb.co/hBWh9cj/logo-mozm-5e95c457fdd1.png',
description: 'Assesses website security posture by analyzing various security headers and practices',
searchLink: 'https://observatory.mozilla.org/analyze/{URL}',
},
]; ];
const makeLink = (resource: any, scanUrl: string | undefined): string => { const makeLink = (resource: any, scanUrl: string | undefined): string => {

View File

@@ -7,7 +7,6 @@ import colors from 'styles/colors';
interface Props { interface Props {
children: ReactNode; children: ReactNode;
title?: string; title?: string;
key?: string;
} }
interface State { interface State {

View File

@@ -224,52 +224,6 @@ const jobNames = [
'carbon', 'carbon',
] as const; ] as const;
interface JobListItemProps {
job: LoadingJob;
showJobDocs: (name: string) => void;
showErrorModal: (name: string, state: LoadingState, timeTaken: number | undefined, error: string, isInfo?: boolean) => void;
barColors: Record<LoadingState, [string, string]>;
}
const getStatusEmoji = (state: LoadingState): string => {
switch (state) {
case 'success':
return '✅';
case 'loading':
return '🔄';
case 'error':
return '❌';
case 'timed-out':
return '⏸️';
case 'skipped':
return '⏭️';
default:
return '❓';
}
};
const JobListItem: React.FC<JobListItemProps> = ({ job, showJobDocs, showErrorModal, barColors }) => {
const { name, state, timeTaken, retry, error } = job;
const actionButton = retry && state !== 'success' && state !== 'loading' ?
<FailedJobActionButton onClick={retry}> Retry</FailedJobActionButton> : null;
const showModalButton = error && ['error', 'timed-out', 'skipped'].includes(state) &&
<FailedJobActionButton onClick={() => showErrorModal(name, state, timeTaken, error, state === 'skipped')}>
{state === 'timed-out' ? '■ Show Timeout Reason' : '■ Show Error'}
</FailedJobActionButton>;
return (
<li key={name}>
<b onClick={() => showJobDocs(name)}>{getStatusEmoji(state)} {name}</b>
<span style={{color: barColors[state][0]}}> ({state})</span>.
<i>{timeTaken && state !== 'loading' ? ` Took ${timeTaken} ms` : ''}</i>
{actionButton}
{showModalButton}
</li>
);
};
export const initialJobs = jobNames.map((job: string) => { export const initialJobs = jobNames.map((job: string) => {
return { return {
name: job, name: job,
@@ -285,9 +239,9 @@ export const calculateLoadingStatePercentages = (loadingJobs: LoadingJob[]): Rec
const stateCount: Record<LoadingState, number> = { const stateCount: Record<LoadingState, number> = {
'success': 0, 'success': 0,
'loading': 0, 'loading': 0,
'timed-out': 0,
'error': 0,
'skipped': 0, 'skipped': 0,
'error': 0,
'timed-out': 0,
}; };
// Count the number of each state // Count the number of each state
@@ -299,9 +253,9 @@ export const calculateLoadingStatePercentages = (loadingJobs: LoadingJob[]): Rec
const statePercentage: Record<LoadingState, number> = { const statePercentage: Record<LoadingState, number> = {
'success': (stateCount['success'] / totalJobs) * 100, 'success': (stateCount['success'] / totalJobs) * 100,
'loading': (stateCount['loading'] / totalJobs) * 100, 'loading': (stateCount['loading'] / totalJobs) * 100,
'timed-out': (stateCount['timed-out'] / totalJobs) * 100,
'error': (stateCount['error'] / totalJobs) * 100,
'skipped': (stateCount['skipped'] / totalJobs) * 100, 'skipped': (stateCount['skipped'] / totalJobs) * 100,
'error': (stateCount['error'] / totalJobs) * 100,
'timed-out': (stateCount['timed-out'] / totalJobs) * 100,
}; };
return statePercentage; return statePercentage;
@@ -399,9 +353,26 @@ const ProgressLoader = (props: { loadStatus: LoadingJob[], showModal: (err: Reac
const barColors: Record<LoadingState | string, [string, string]> = { const barColors: Record<LoadingState | string, [string, string]> = {
'success': isDone ? makeBarColor(colors.primary) : makeBarColor(colors.success), 'success': isDone ? makeBarColor(colors.primary) : makeBarColor(colors.success),
'loading': makeBarColor(colors.info), 'loading': makeBarColor(colors.info),
'skipped': makeBarColor(colors.warning),
'error': makeBarColor(colors.danger), 'error': makeBarColor(colors.danger),
'timed-out': makeBarColor(colors.warning), 'timed-out': makeBarColor(colors.neutral),
'skipped': makeBarColor(colors.neutral), };
const getStatusEmoji = (state: LoadingState): string => {
switch (state) {
case 'success':
return '✅';
case 'loading':
return '🔄';
case 'skipped':
return '⏭️';
case 'error':
return '❌';
case 'timed-out':
return '⏸️';
default:
return '❓';
}
}; };
const showErrorModal = (name: string, state: LoadingState, timeTaken: number | undefined, error: string, isInfo?: boolean) => { const showErrorModal = (name: string, state: LoadingState, timeTaken: number | undefined, error: string, isInfo?: boolean) => {
@@ -445,9 +416,20 @@ const ProgressLoader = (props: { loadStatus: LoadingJob[], showModal: (err: Reac
<Details> <Details>
<summary>Show Details</summary> <summary>Show Details</summary>
<ul> <ul>
{loadStatus.map((job: LoadingJob) => ( {
<JobListItem key={job.name} job={job} showJobDocs={props.showJobDocs} showErrorModal={showErrorModal} barColors={barColors} /> loadStatus.map(({ name, state, timeTaken, retry, error }: LoadingJob) => {
))} return (
<li key={name}>
<b onClick={() => props.showJobDocs(name)}>{getStatusEmoji(state)} {name}</b>
<span style={{color : barColors[state][0]}}> ({state})</span>.
<i>{(timeTaken && state !== 'loading') ? ` Took ${timeTaken} ms` : '' }</i>
{ (retry && state !== 'success' && state !== 'loading') && <FailedJobActionButton onClick={retry}> Retry</FailedJobActionButton> }
{ (error && state === 'error') && <FailedJobActionButton onClick={() => showErrorModal(name, state, timeTaken, error)}> Show Error</FailedJobActionButton> }
{ (error && state === 'skipped') && <FailedJobActionButton onClick={() => showErrorModal(name, state, timeTaken, error, true)}> Show Reason</FailedJobActionButton> }
</li>
);
})
}
</ul> </ul>
{ loadStatus.filter((val: LoadingJob) => val.state === 'error').length > 0 && { loadStatus.filter((val: LoadingJob) => val.state === 'error').length > 0 &&
<p className="error"> <p className="error">

View File

@@ -35,18 +35,13 @@ const useMotherOfAllHooks = <ResultType = any>(params: UseIpAddressProps<ResultT
const [result, setResult] = useState<ResultType>(); const [result, setResult] = useState<ResultType>();
// Fire off the HTTP fetch request, then set results and update loading / error state // Fire off the HTTP fetch request, then set results and update loading / error state
const doTheFetch = () => { const doTheFetch = () => {
return fetchRequest() return fetchRequest()
.then((res: any) => { .then((res: any) => {
if (!res) { // No response :( if (!res) { // No response :(
updateLoadingJobs(jobId, 'error', 'No response', reset); updateLoadingJobs(jobId, 'error', res.error || 'No response', reset);
} else if (res.error) { // Response returned an error message } else if (res.error) { // Response returned an error message
if (res.error.includes("timed-out")) { // Specific handling for timeout errors
updateLoadingJobs(jobId, 'timed-out', res.error, reset);
} else {
updateLoadingJobs(jobId, 'error', res.error, reset); updateLoadingJobs(jobId, 'error', res.error, reset);
}
} else if (res.skipped) { // Response returned a skipped message } else if (res.skipped) { // Response returned a skipped message
updateLoadingJobs(jobId, 'skipped', res.skipped, reset); updateLoadingJobs(jobId, 'skipped', res.skipped, reset);
} else { // Yay, everything went to plan :) } else { // Yay, everything went to plan :)

View File

@@ -101,18 +101,6 @@ const Section = styled(StyledCard)`
} }
`; `;
const SponsorshipContainer = styled.div`
display: flex;
justify-content: space-between;
gap: 1rem;
flex-wrap: wrap;
align-items: center;
line-height: 1.5rem;
img {
border-radius: 4px;
}
`;
const makeAnchor = (title: string): string => { const makeAnchor = (title: string): string => {
return title.toLowerCase().replace(/[^\w\s]|_/g, "").replace(/\s+/g, "-"); return title.toLowerCase().replace(/[^\w\s]|_/g, "").replace(/\s+/g, "-");
}; };
@@ -123,7 +111,7 @@ const About = (): JSX.Element => {
<AboutContainer> <AboutContainer>
<Nav> <Nav>
<HeaderLinkContainer> <HeaderLinkContainer>
<a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check"><Button>View on GitHub</Button></a> <a href="https://github.com/lissy93/web-check"><Button>View on GitHub</Button></a>
</HeaderLinkContainer> </HeaderLinkContainer>
</Nav> </Nav>
@@ -133,32 +121,12 @@ const About = (): JSX.Element => {
<p key={index}>{para}</p> <p key={index}>{para}</p>
))} ))}
<hr /> <hr />
<SponsorshipContainer>
<p> <p>
Web-Check is kindly sponsored Web-Check is developed and maintained by <a href="https://aliciasykes.com">Alicia Sykes</a>.
by <a target="_blank" rel="noreferrer" href="https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh"> It's licensed under the <a href="https://github.com/Lissy93/web-check/blob/master/LICENSE">MIT license</a>,
Terminal Trove
</a>
<br />
The $HOME of all things in the terminal.
<br />
<small>
<a target="_blank" rel="noreferrer" href="https://terminaltrove.com/newsletter?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
Find your next CLI / TUI tool, and get updates to your inbox
</a>
</small>
</p>
<a target="_blank" rel="noreferrer" href="https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
<img width="300" alt="Terminal Trove" src="https://i.ibb.co/T1KzVmR/terminal-trove-green.png" />
</a>
</SponsorshipContainer>
<hr />
<p>
Web-Check is developed and maintained by <a target="_blank" rel="noreferrer" href="https://aliciasykes.com">Alicia Sykes</a>.
It's licensed under the <a target="_blank" rel="noreferrer" href="https://github.com/Lissy93/web-check/blob/master/LICENSE">MIT license</a>,
and is completely free to use, modify and distribute in both personal and commercial settings.<br /> and is completely free to use, modify and distribute in both personal and commercial settings.<br />
Source code and self-hosting docs are available on <a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check">GitHub</a>. Source code and self-hosting docs are available on <a href="https://github.com/lissy93/web-check">GitHub</a>.
If you've found this service useful, consider <a target="_blank" rel="noreferrer" href="https://github.com/sponsors/Lissy93">sponsoring me</a> from $1/month, If you've found this service useful, consider <a href="https://github.com/sponsors/Lissy93">sponsoring me</a> from $1/month,
to help with the ongoing hosting and development costs. to help with the ongoing hosting and development costs.
</p> </p>
</Section> </Section>
@@ -211,79 +179,6 @@ const About = (): JSX.Element => {
))} ))}
</Section> </Section>
<Heading as="h2" size="medium" color={colors.primary}>Deploy your own Instance</Heading>
<Section>
<p>Web-Check is designed to be easily self-hosted.</p>
<Heading as="h3" size="small" color={colors.primary}>Option #1 - Netlify</Heading>
<p>Click the button below to deploy to Netlify</p>
<a target="_blank" rel="noreferrer" href="https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check">
<img src="https://www.netlify.com/img/deploy/button.svg" alt="Deploy to Netlify" />
</a>
<Heading as="h3" size="small" color={colors.primary}>Option #2 - Vercel</Heading>
<p>Click the button below to deploy to Vercel</p>
<a target="_blank" rel="noreferrer" href="https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flissy93%2Fweb-check&project-name=web-check&repository-name=web-check-fork&demo-title=Web-Check%20Demo&demo-description=Check%20out%20web-check.xyz%20to%20see%20a%20live%20demo%20of%20this%20application%20running.&demo-url=https%3A%2F%2Fweb-check.xyz&demo-image=https%3A%2F%2Fraw.githubusercontent.com%2FLissy93%2Fweb-check%2Fmaster%2F.github%2Fscreenshots%2Fweb-check-screenshot10.png">
<img src="https://vercel.com/button" alt="Deploy with Vercel" />
</a>
<Heading as="h3" size="small" color={colors.primary}>Option #3 - Docker</Heading>
<p>
A Docker container is published to <a target="_blank" rel="noreferrer" href="https://hub.docker.com/r/lissy93/web-check">DockerHub</a>
<br />
Run this command, then open <code>localhost:3000</code>
<pre>docker run -p 3000:3000 lissy93/web-check</pre>
</p>
<Heading as="h3" size="small" color={colors.primary}>Option #4 - Manual</Heading>
<pre>
git clone https://github.com/Lissy93/web-check.git<br />
cd web-check # Move into the project directory<br />
yarn install # Install dependencies<br />
yarn build # Build the app for production<br />
yarn serve # Start the app (API and GUI)<br />
</pre>
<Heading as="h3" size="small" color={colors.primary}>Further Docs</Heading>
<p>
More detailed installation and setup instructions can be found in the
GitHub repository - <a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check#readme">github.com/lissy93/web-check</a>
</p>
<Heading as="h3" size="small" color={colors.primary}>Configuring</Heading>
<p>
There are some optional environmental variables you can specify to give you access to some additional Web-Checks.
See the README for full list of options.
</p>
<ul>
<li>
<code>GOOGLE_CLOUD_API_KEY</code>
: <a target="_blank" rel="noreferrer" href="https://cloud.google.com/api-gateway/docs/authenticate-api-keys">A Google API key</a>
<i> Used to return quality metrics for a site</i>
</li>
<li>
<code>REACT_APP_SHODAN_API_KEY</code>
: <a target="_blank" rel="noreferrer" href="https://account.shodan.io/">A Shodan API key</a>
<i> To show associated hosts for a domain</i>
</li>
<li>
<code>REACT_APP_WHO_API_KEY</code>
: <a target="_blank" rel="noreferrer" href="https://whoapi.com/">A WhoAPI key</a>
<i> Allows for more comprehensive WhoIs records</i>
</li>
</ul>
{/*
**Configuration Settings**:
- `CHROME_PATH` (e.g. `/usr/bin/chromium`) - The path the the Chromium executable
- `PORT` (e.g. `3000`) - Port to serve the API, when running server.js
- `DISABLE_GUI` (e.g. `false`) - Disable the GUI, and only serve the API
- `API_TIMEOUT_LIMIT` (e.g. `10000`) - The timeout limit for API requests, in milliseconds
- `REACT_APP_API_ENDPOINT` (e.g. `/api`) - The endpoint for the API (can be local or remote)</p> */}
</Section>
<Heading as="h2" size="medium" color={colors.primary}>API Documentation</Heading> <Heading as="h2" size="medium" color={colors.primary}>API Documentation</Heading>
<Section> <Section>
{/* eslint-disable-next-line*/} {/* eslint-disable-next-line*/}
@@ -302,11 +197,11 @@ const About = (): JSX.Element => {
<Section> <Section>
<Heading as="h3" size="small" color={colors.primary}>License</Heading> <Heading as="h3" size="small" color={colors.primary}>License</Heading>
<b> <b>
<a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check">Web-Check</a> is distributed under the MIT license, <a href="https://github.com/lissy93/web-check">Web-Check</a> is distributed under the MIT license,
© <a target="_blank" rel="noreferrer" href="https://aliciasykes.com">Alicia Sykes</a> { new Date().getFullYear()} © <a href="https://aliciasykes.com">Alicia Sykes</a> { new Date().getFullYear()}
</b> </b>
<br /> <br />
<small>For more info, see <a target="_blank" rel="noreferrer" href="https://tldrlegal.com/license/mit-license">TLDR Legal MIT</a></small> <small>For more info, see <a href="https://tldrlegal.com/license/mit-license">TLDR Legal MIT</a></small>
<pre>{license}</pre> <pre>{license}</pre>
<hr /> <hr />
<Heading as="h3" size="small" color={colors.primary}>Fair Use</Heading> <Heading as="h3" size="small" color={colors.primary}>Fair Use</Heading>
@@ -323,6 +218,18 @@ const About = (): JSX.Element => {
Neither your IP address, browser/OS/hardware info, nor any other data will ever be collected or logged. Neither your IP address, browser/OS/hardware info, nor any other data will ever be collected or logged.
(You may verify this yourself, either by inspecting the source code or the using developer tools) (You may verify this yourself, either by inspecting the source code or the using developer tools)
</p> </p>
<hr />
<Heading as="h3" size="small" color={colors.primary}>Support</Heading>
<p>
If you've found something that doesn't work as expected, or would like to ask any questions,
you can open a ticket at <a href="https://github.com/lissy93/web-check/issues">github.com/lissy93/web-check/issues</a>
</p>
<hr />
<Heading as="h3" size="small" color={colors.primary}>Sponsor</Heading>
<p>
If you've found this service useful, consider sponsoring me on
GitHub - <a href="https://github.com/sponsors/Lissy93">github.com/sponsors/Lissy93</a> 💖
</p>
</Section> </Section>
</AboutContainer> </AboutContainer>
<Footer /> <Footer />

View File

@@ -20,7 +20,7 @@ const HomeContainer = styled.section`
align-items: center; align-items: center;
height: 100%; height: 100%;
font-family: 'PTMono'; font-family: 'PTMono';
padding: 1.5rem 1rem 4rem 1rem; padding: 0 1rem;
footer { footer {
z-index: 1; z-index: 1;
} }
@@ -34,52 +34,10 @@ const UserInputMain = styled.form`
z-index: 5; z-index: 5;
margin: 1rem; margin: 1rem;
width: calc(100% - 2rem); width: calc(100% - 2rem);
max-width: 60rem; max-width: 50rem;
z-index: 2; z-index: 2;
`; `;
const SponsorCard = styled.div`
background: ${colors.backgroundLighter};
box-shadow: 4px 4px 0px ${colors.bgShadowColor};
border-radius: 8px;
padding: 1rem;
z-index: 5;
margin: 1rem;
width: calc(100% - 2rem);
max-width: 60rem;
z-index: 2;
.inner {
display: flex;
justify-content: space-between;
flex-wrap: wrap;
gap: 1rem;
p {
margin: 0.25rem 0;
}
}
a {
color: ${colors.textColor};
}
img {
border-radius: 0.25rem;
box-shadow: 2px 2px 0px ${colors.fgShadowColor};
transition: box-shadow 0.2s;
margin: 0 auto;
display: block;
width: 200px;
&:hover {
box-shadow: 4px 4px 0px ${colors.fgShadowColor};
}
&:active {
box-shadow: -2px -2px 0px ${colors.fgShadowColor};
}
}
.cta {
font-size: 0.78rem;
a { color: ${colors.primary}; }
}
`;
// const FindIpButton = styled.a` // const FindIpButton = styled.a`
// margin: 0.5rem; // margin: 0.5rem;
// cursor: pointer; // cursor: pointer;
@@ -97,7 +55,7 @@ const ErrorMessage = styled.p`
const SiteFeaturesWrapper = styled(StyledCard)` const SiteFeaturesWrapper = styled(StyledCard)`
margin: 1rem; margin: 1rem;
width: calc(100% - 2rem); width: calc(100% - 2rem);
max-width: 60rem; max-width: 50rem;
z-index: 2; z-index: 2;
.links { .links {
display: flex; display: flex;
@@ -215,28 +173,6 @@ const Home = (): JSX.Element => {
{ errorMsg && <ErrorMessage>{errorMsg}</ErrorMessage>} { errorMsg && <ErrorMessage>{errorMsg}</ErrorMessage>}
<Button styles="width: calc(100% - 1rem);" size="large" onClick={submit}>Analyze!</Button> <Button styles="width: calc(100% - 1rem);" size="large" onClick={submit}>Analyze!</Button>
</UserInputMain> </UserInputMain>
<SponsorCard>
<Heading as="h2" size="small" color={colors.primary}>Sponsored by</Heading>
<div className="inner">
<p>
<a target="_blank" rel="noreferrer" href="https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
Terminal Trove
</a> - The $HOME of all things in the terminal.
<br />
<span className="cta">
Get updates on the latest CLI/TUI tools via
the <a target="_blank" rel="noreferrer" className="cta" href="https://terminaltrove.com/newsletter?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
Terminal Trove newsletter
</a>
</span>
</p>
<a target="_blank" rel="noreferrer" href="https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=web-check&utm_source=wcgh">
<img width="120" alt="Terminal Trove" src="https://i.ibb.co/NKtYjJ1/terminal-trove-web-check.png" />
</a>
</div>
</SponsorCard>
<SiteFeaturesWrapper> <SiteFeaturesWrapper>
<div className="features"> <div className="features">
<Heading as="h2" size="small" color={colors.primary}>Supported Checks</Heading> <Heading as="h2" size="small" color={colors.primary}>Supported Checks</Heading>
@@ -246,10 +182,10 @@ const Home = (): JSX.Element => {
</ul> </ul>
</div> </div>
<div className="links"> <div className="links">
<a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check" title="Check out the source code and documentation on GitHub, and get support or contribute"> <a href="https://github.com/lissy93/web-check" title="Check out the source code and documentation on GitHub, and get support or contribute">
<Button>View on GitHub</Button> <Button>View on GitHub</Button>
</a> </a>
<a target="_blank" rel="noreferrer" href="https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check" title="Deploy your own private or public instance of Web-Check to Netlify"> <a href="https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check" title="Deploy your own private or public instance of Web-Check to Netlify">
<Button>Deploy your own</Button> <Button>Deploy your own</Button>
</a> </a>
<a href="/about#api-documentation" title="View the API documentation, to use Web-Check programmatically"> <a href="/about#api-documentation" title="View the API documentation, to use Web-Check programmatically">

View File

@@ -57,7 +57,7 @@ const NotFound = (): JSX.Element => {
<HeaderLinkContainer> <HeaderLinkContainer>
<a href="/"><Button>Back to Homepage</Button></a> <a href="/"><Button>Back to Homepage</Button></a>
</HeaderLinkContainer> </HeaderLinkContainer>
<a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check">Report Issue</a> <a href="https://github.com/lissy93/web-check">Report Issue</a>
</NotFoundInner> </NotFoundInner>
</AboutContainer> </AboutContainer>
<Footer isFixed={true} /> <Footer isFixed={true} />

View File

@@ -208,24 +208,12 @@ const Results = (): JSX.Element => {
console.log( console.log(
`%cFetch Error - ${job}%c\n\n${timeString}%c The ${job} job failed ` `%cFetch Error - ${job}%c\n\n${timeString}%c The ${job} job failed `
+`after ${timeTaken}ms, with the following error:%c\n${error}`, +`after ${timeTaken}ms, with the following error:%c\n${error}`,
`background: ${colors.danger}; color:${colors.background}; padding: 4px 8px; font-size: 16px;`, `background: ${colors.danger}; padding: 4px 8px; font-size: 16px;`,
`font-weight: bold; color: ${colors.danger};`, `font-weight: bold; color: ${colors.danger};`,
`color: ${colors.danger};`, `color: ${colors.danger};`,
`color: ${colors.warning};`, `color: ${colors.warning};`,
); );
} }
if (newState === 'timed-out') {
console.log(
`%cFetch Timeout - ${job}%c\n\n${timeString}%c The ${job} job timed out `
+`after ${timeTaken}ms, with the following error:%c\n${error}`,
`background: ${colors.info}; color:${colors.background}; padding: 4px 8px; font-size: 16px;`,
`font-weight: bold; color: ${colors.info};`,
`color: ${colors.info};`,
`color: ${colors.warning};`,
);
}
return newJobs; return newJobs;
}); });
}); });
@@ -237,9 +225,8 @@ const Results = (): JSX.Element => {
.then(data => resolve(data)) .then(data => resolve(data))
.catch(error => resolve( .catch(error => resolve(
{ error: `Failed to get a valid response 😢\n` { error: `Failed to get a valid response 😢\n`
+ 'This is likely due the target not exposing the required data, ' + `This is likely due the target not exposing the required data, `
+ 'or limitations in imposed by the infrastructure this instance ' + `or limitations in how Netlify executes lambda functions, such as the 10-sec timeout.\n\n`
+ 'of Web Check is running on.\n\n'
+ `Error info:\n${error}`} + `Error info:\n${error}`}
)); ));
}); });
@@ -301,7 +288,7 @@ const Results = (): JSX.Element => {
addressInfo: { address, addressType, expectedAddressTypes: urlTypeOnly }, addressInfo: { address, addressType, expectedAddressTypes: urlTypeOnly },
fetchRequest: () => fetch(`${api}/quality?url=${address}`) fetchRequest: () => fetch(`${api}/quality?url=${address}`)
.then(res => parseJson(res)) .then(res => parseJson(res))
.then(res => res?.lighthouseResult || { error: res.error || 'No Data' }), .then(res => res?.lighthouseResult || { error: 'No Data'}),
}); });
// Get the technologies used to build site, using Wappalyzer // Get the technologies used to build site, using Wappalyzer
@@ -870,13 +857,13 @@ const Results = (): JSX.Element => {
<Nav> <Nav>
{ address && { address &&
<Heading color={colors.textColor} size="medium"> <Heading color={colors.textColor} size="medium">
{ addressType === 'url' && <a target="_blank" rel="noreferrer" href={address}><img width="32px" src={`https://icon.horse/icon/${makeSiteName(address)}`} alt="" /></a> } { addressType === 'url' && <a href={address}><img width="32px" src={`https://icon.horse/icon/${makeSiteName(address)}`} alt="" /></a> }
{makeSiteName(address)} {makeSiteName(address)}
</Heading> </Heading>
} }
</Nav> </Nav>
<ProgressBar loadStatus={loadingJobs} showModal={showErrorModal} showJobDocs={showInfo} /> <ProgressBar loadStatus={loadingJobs} showModal={showErrorModal} showJobDocs={showInfo} />
{ address?.includes(window?.location?.hostname || 'web-check.xyz') && <SelfScanMsg />} { address?.includes(window?.location?.hostname || 'web-check.as93.net') && <SelfScanMsg />}
<Loader show={loadingJobs.filter((job: LoadingJob) => job.state !== 'loading').length < 5} /> <Loader show={loadingJobs.filter((job: LoadingJob) => job.state !== 'loading').length < 5} />
<FilterButtons>{ showFilters ? <> <FilterButtons>{ showFilters ? <>
<div className="one-half"> <div className="one-half">
@@ -907,7 +894,7 @@ const Results = (): JSX.Element => {
<a href="#view-download-raw-data"><span className="toggle-filters">Export Data</span></a> <a href="#view-download-raw-data"><span className="toggle-filters">Export Data</span></a>
<a href="/about"><span className="toggle-filters">Learn about the Results</span></a> <a href="/about"><span className="toggle-filters">Learn about the Results</span></a>
<a href="/about#additional-resources"><span className="toggle-filters">More tools</span></a> <a href="/about#additional-resources"><span className="toggle-filters">More tools</span></a>
<a target="_blank" rel="noreferrer" href="https://github.com/lissy93/web-check"><span className="toggle-filters">View GitHub</span></a> <a href="https://github.com/lissy93/web-check"><span className="toggle-filters">View GitHub</span></a>
</div> </div>
) } ) }
</FilterButtons> </FilterButtons>
@@ -923,7 +910,7 @@ const Results = (): JSX.Element => {
&& title.toLowerCase().includes(searchTerm.toLowerCase()) && title.toLowerCase().includes(searchTerm.toLowerCase())
&& (result && !result.error); && (result && !result.error);
return show ? ( return show ? (
<ErrorBoundary title={title} key={`eb-${index}`}> <ErrorBoundary title={title}>
<Component <Component
key={`${title}-${index}`} key={`${title}-${index}`}
data={{...result}} data={{...result}}

View File

@@ -529,7 +529,7 @@ and the web of connections within a site's architecture.
The results can also help optimizing server responses, configuring redirects, The results can also help optimizing server responses, configuring redirects,
managing cookies, or fine-tuning DNS records for your site.`, managing cookies, or fine-tuning DNS records for your site.`,
`So, whether you're a developer, system administrator, security researcher, penetration `So, weather you're a developer, system administrator, security researcher, penetration
tester or are just interested in discovering the underlying technologies of a given site tester or are just interested in discovering the underlying technologies of a given site
- I'm sure you'll find this a useful addition to your toolbox.`, - I'm sure you'll find this a useful addition to your toolbox.`,
]; ];
@@ -557,7 +557,7 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
export const supportUs = [ export const supportUs = [
"Web-Check is free to use without restriction.", "Web-Check is free to use without restriction.",
"All the code is open source, so you're also free to deploy your own instance, as well as fork, modify and distribute the code in both private and commercial settings.", "All the code is open source, so you're also free to deploy your own instance, as well as fork, modify and distribute the code in both private and commerical settings.",
"Running web-check does cost me a small amount of money each month, so if you're finding the app useful, consider <a href='https://github.com/sponsors/Lissy93'>sponsoring me on GitHub</a> if you're able to. Even just $1 or $2/month would be a huge help in supporting the ongoing project running costs.", "Running web-check does cost me a small amount of money each month, so if you're finding the app useful, consider <a href='https://github.com/sponsors/Lissy93'>sponsoring me on GitHub</a> if you're able to. Even just $1 or $2/month would be a huge help in supporting the ongoing project running costs.",
"Otherwise, there are other ways you can help out, like submitting or reviewing a pull request to the <a href='https://github.com/Lissy93/web-check'>GitHub repo</a>, upvoting us on <a href='https://www.producthunt.com/posts/web-check'>Product Hunt</a>, or by sharing with your network.", "Otherwise, there are other ways you can help out, like submitting or reviewing a pull request to the <a href='https://github.com/Lissy93/web-check'>GitHub repo</a>, upvoting us on <a href='https://www.producthunt.com/posts/web-check'>Product Hunt</a>, or by sharing with your network.",
"But don't feel obliged to do anything, as this app (and all my other projects) will always remain 100% free and open source, and I will do my best to ensure the managed instances remain up and available for as long as possible :)", "But don't feel obliged to do anything, as this app (and all my other projects) will always remain 100% free and open source, and I will do my best to ensure the managed instances remain up and available for as long as possible :)",

View File

@@ -1,7 +0,0 @@
{
"functions": {
"api/*.js": {
"maxDuration": 10
}
}
}

4883
yarn.lock

File diff suppressed because it is too large Load Diff